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ABSTRACT 


We verified the USAF World Wide Merged Cloud Analysis (WWMCA) against 
observations from the Cloudsat atmospheric sounder. We analyzed WWMCA data for 
2010 for two regions that are of high interest to the national intelligence community 
and that differ in their meteorological characteristics. The two regions covered were: 
(a) much of southwest Asia; and (b) much of western Russia and the Barents Sea. We 
analyzed WWMCA performance according to four criteria: (1) type of cloud event 
(Definite-Cloud, Probable-Cloud, and No-Cloud); (2) geographic region; (3) time of day; 
and (4) time of year. We measured WWMCA performance using contingency table 
metrics and found marked differences in performance for the four criteria. In particular, 
WWMCA tended to perform better in analyzing: (a) No-Cloud and Definite-Cloud events 
than Probable-Cloud events; (b) the lower latitude region than the higher latitude region; 
and (c) persistent cloud events than variable cloud events. Our Heidke skill scores 
indicated that WWMCA performance was, in general, moderately better than that of a 
random set of analyses. Overall, WWMCA performance was problematic, given that 
WWMCA is a near real-time analysis product and is designed to initiate short lead time 
cloud forecasts used by the intelligence community. 
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A schematic illustration of the approach being used at NPS to 
quantitatively assess the impacts of cloud analyses, cloud forecasts, and 
actual clouds on operational planning and operational outcomes. The 
upper left portion of the figure summarizes the approach being used in this 
study to verify WWMCA cloud analyses. The upper right portion of the 
figure summarizes the approach being used in a companion NPS study to 
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Schematic overview of the Cloud Depiction Forecast System II. 
Observational data are received from meteorological satellites, 
conventional observations, and global analyses from various models. 
These data are merged into one global cloud analysis, called WWMCA, 
which is used to initiate cloud forecast models. Quality control is 
performed on the cloud analysis and model output products by a 


forecaster. Figure from HQ AFWA/DNXM (2011).10 

Worldwide Merge Cloud Analysis file format. Figure from HQ 

AFWA/DNXM (2011).12 

Depiction of the satellites that make up the “A-train”. Each satellite has 
its name and time of equator crossing listed. Note that CALIPSO trails 
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CALIPSO. From NASA Facts-A-Train 2003.13 

The instantaneous field of view (IFOV) for the Cloudsat cloud profiling 

radar (CPR). From Cloudsat Product Handbook (2011).15 

A visualization of Cloudsat data for one segment of a granule. The top 
pane is the 1A Aux data and contains the date and start time of the 


granule, and the starting/ending latitude-longitude and times of that 
segment. The middle panel is the 2B-GeoProf cloud masking, which is 
color coded for masking values. The bottom panel is an Aqua MODIS 
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(2011).17 

Location of WWMCA boxes 22 and 29, shown by the white southern and 
northern boxes, respectively. These boxes appear skewed in the image 
due to the map projection. Box 22 includes much or all of Iraq, Iran, 
Afghanistan, and Pakistan. Box 29 includes much of western Russia and 

the Barents Sea. Base image from Google Earth (2011).18 

The WWMCA grid is organized into 128 square boxes (64 boxes per 
hemisphere) with each box containing 128 cells. The cells have unique i-j 
coordinates that describe the latitude and longitude of the cell. The cells 
are 24 km long on a side (true at 60° latitude). The yellow shaded boxes 
are the two WWMCA boxes that we focused on in this study. The 


IX 










Figure 9. 
Figure 10. 

Figure 11. 


Figure 12. 


Figure 13. 


Figure 14. 


Figure 15. 


northern box is WWMCA Box 29 and the southern one is Box 22. Base 

figures from 16 th Weather Squadron (2011).20 
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Cloudsat datasets for analysis.22 
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amount is 63%.25 

Number of 2 x 2 contingency table occurrences in January and April 2010 
for WWMCA Box 22 for all three cloud cover events: Definite-Cloud 
(blue), Probable-Cloud (red), and No-Cloud (green). On the horizontal 
axes: A indicates Hits, B indicates False Alarms, C indicates Misses, and 
D indicates Correct Rejections. The top panels shows the daytime 
occurrences, the middle panels shows the nighttime occurrences, and the 
bottom panels show the results for both daytime and nighttime 

occurrences. Results based on using a Cloudsat cloud mask value of 30 

for the cloud/no-cloud threshold.36 

Number of 2 x 2 contingency table occurrences in July and October 2010 
for WWMCA Box 22 for all three cloud cover events: Definite-Cloud 
(blue), Probable-Cloud (red), and No-Cloud (green). On the horizontal 
axes: A indicates Hits, B indicates False Alarms, C indicates Misses, and 
D indicates Correct Rejections. The top panels shows the daytime 
occurrences, the middle panels shows the nighttime occurrences, and the 
bottom panels show the results for both daytime and nighttime 

occurrences. Results based on using a Cloudsat cloud mask value of 30 

for the cloud/no-cloud threshold.37 

Marginal distributions of cloud cover events for WWMCA and Cloudsat 
for WWMCA Box 22: Definite-Cloud, Probable-Cloud, and No-Cloud). 

The calculation of the marginal distributions is shown in Table 7. The top 
panel shows the results for daytime data, the middle panel for nighttime 
data, and the bottom panel for both daytime and nighttime data. Results 
based on using a Cloudsat cloud mask value of 30 for the cloud/no-cloud 

threshold.38 

WWMCA probability of detection (POD), or hit rate (H), for WWMCA 
Box 22 for all three cloud cover events: Definite-Cloud (blue), Probable- 
Cloud (red), and No-Cloud (green). The top panel shows the results for 
daytime data, the middle panel for nighttime data, and the bottom panel 
for both daytime and nighttime data. Results based on using a Cloudsat 
cloud mask value of 30 for the cloud/no-cloud threshold.40 
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WWMCA false alarm ratios for WWMCA Box 22 for all three cloud 
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for the cloud/no-cloud threshold.42 

WWMCA proportion correct (PC) for WWMCA Box 22 for all three 
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nighttime data. Results based on using a Cloudsat cloud mask value of 30 

for the cloud/no-cloud threshold.45 

WWMCA bias (B) for WWMCA Box 22 for all three cloud cover events: 
Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud (green). The 
top panel shows the results for daytime data, the middle panel for 
nighttime data, and the bottom panel for both daytime and nighttime data. 

The heavy dark line at B = 1 indicates an unbiased analysis. Values 
greater (less) than 1 indicate a cloud cover event that was over (under) 
analyzed. Results based on using a Cloudsat cloud mask value of 30 for 

the cloud/no-cloud threshold.47 

WWMCA Heidke skill score (HSS) for WWMCA Box 22 for all three 
cloud cover events: Definite-Cloud (blue), Probable-Cloud (red), and No- 
Cloud (green). The top panel shows the results for daytime data, the 
middle panel for nighttime data, and the bottom panel for both daytime 
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the WWMCA is equivalent to reference analyses (random analyses 
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analysis would have a value of one. Negative values indicate WWMCA 
performs worse than reference analyses. Results based on using a 

Cloudsat cloud mask value of 30 for the cloud/no-cloud threshold.49 

Number of 2 x 2 contingency table occurrences in January and April 2010 
for WWMCA Box 29 for all three cloud cover events: Definite-Cloud 
(blue), Probable-Cloud (red), and No-Cloud (green). On the horizontal 
axes: A indicates Hits, B indicates False Alarms, C indicates Misses, and 
D indicates Correct Rejections. The top panels shows the daytime 
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occurrences. Results based on using a Cloudsat cloud mask value of 30 
for the cloud/no-cloud threshold.51 
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axes: A indicates Hits, B indicates False Alarms, C indicates Misses, and 
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bottom panels show the results for both daytime and nighttime 
occurrences. Results based on using a Cloudsat cloud mask value of 30 

for the cloud/no-cloud threshold.52 

Marginal distributions of cloud cover events for WWMCA and Cloudsat 
for WWMCA Box 29: Definite-Cloud, Probable-Cloud, and No-Cloud). 

The calculation of the marginal distributions is shown in Table 7. The top 
panel shows the results for daytime data, the middle panel for nighttime 
data, and the bottom panel for both daytime and nighttime data. Results 
based on using a Cloudsat cloud mask value of 30 for the cloud/no-cloud 

threshold.53 

WWMCA probability of detection (POD), or hit rate (H), for WWMCA 
Box 29 for all three cloud cover events: Definite-Cloud (blue), Probable- 
Cloud (red), and No-Cloud (green). The top panel shows the results for 
daytime data, the middle panel for nighttime data, and the bottom panel 
for both daytime and nighttime data. Results based on using a Cloudsat 

cloud mask value of 30 for the cloud/no-cloud threshold.54 

WWMCA false alarm ratios for WWMCA Box 29 for all three cloud 
cover events: Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud 
(green). The top panel shows the results for daytime data, the middle 
panel for nighttime data, and the bottom panel for both daytime and 
nighttime data. Results based on using a Cloudsat cloud mask value of 30 

for the cloud/no-cloud threshold.55 

WWMCA proportion correct (PC) for WWMCA Box 29 for all three 
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Cloud (green). The top panel shows the results for daytime data, the 
middle panel for nighttime data, and the bottom panel for both daytime 
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greater (less) than 1 indicate a cloud cover event that was over (under) 
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Number of occurrences in January and April 2010 for WWMCA Box 22 
for all three cloud cover events (cloud, probable cloud, and no-cloud) 
using a Cloudsat cloud mask value of 20 for cloud/no-cloud threshold. 

The “cloud” event (blue) represents the “definite cloud” event. The 
horizontal axes are the occurrence categories: A—Hits; B—False Alarms; 

C—Misses; and D—Correct Rejections. The top panels are the 
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occurrences.87 
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for WWMCA Box 22 for all three cloud cover events: Definite-Cloud 
(blue), Probable-Cloud (red), and No-Cloud (green). On the horizontal 
axes: A indicates Hits, B indicates False Alarms, C indicates Misses, and 
D indicates Correct Rejections. The top panels shows the daytime 
occurrences, the middle panels shows the nighttime occurrences, and the 
bottom panels show the results for both daytime and nighttime 
occurrences. Results based on using a Cloudsat cloud mask value of 20 

for the cloud/no-cloud threshold.88 

Marginal distributions of cloud cover events for WWMCA and Cloudsat 
for WWMCA Box 22: Definite-Cloud, Probable-Cloud, and No-Cloud). 
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(green). The top panel shows the results for daytime data, the middle 
panel for nighttime data, and the bottom panel for both daytime and 
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nighttime data. Results based on using a Cloudsat cloud mask value of 20 

for the cloud/no-cloud threshold.102 

Figure 47. WWMCA bias (B) for WWMCA Box 29 for all three cloud cover events: 

Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud (green). The 
top panel shows the results for daytime data, the middle panel for 
nighttime data, and the bottom panel for both daytime and nighttime data. 

The heavy dark line at B = 1 indicates an unbiased analysis. Values 
greater (less) than 1 indicate a cloud cover event that was over (under) 
analyzed. Results based on using a Cloudsat cloud mask value of 20 for 

the cloud/no-cloud threshold.103 

Figure 48. WWMCA Heidke skill score (HSS) for WWMCA Box 29 for all three 
cloud cover events: Definite-Cloud (blue), Probable-Cloud (red), and No- 
Cloud (green). The top panel shows the results for daytime data, the 
middle panel for nighttime data, and the bottom panel for both daytime 
and nighttime data. The heavy dark line at the zero value indicates that 
the WWMCA is equivalent to reference analyses (random analyses 
statistically independent of observations). A perfect WWMCA cloud 
analysis would have a value of one. Negative values indicate WWMCA 
performs worse than reference analyses. Results based on using a 
Cloudsat cloud mask value of 20 for the cloud/no-cloud threshold.104 
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I. INTRODUCTION 


A. IMPORTANCE OF CLOUD ANALYSES AND FORECASTS 

Cloud analysis and forecasts are essential to the overhead imagery mission. 
Imagery satellites play a vital role in the national intelligence mission. They are low 
density/high demand assets that are quite sensitive to cloud masking of targets. Cloud 
cover impacts satellite imagery collection by obscuring sensors operating in the visual 
and infrared (IR) channels. Accurate cloud forecasts assist in target selection, the asset 
scheduling, and help reduce the need to reshoot targets. To improve cloud forecasts, a 
firm verification of the cloud analyses that go into cloud forecasts, and of the forecasts 
themselves, is needed. 

Cloud analysis verification is very difficult, because: (1) clouds can form and 
dissipate in a relatively short period; and (2) clouds reside at various levels in the 
atmosphere and are advected by winds via nonlinear processes. Satellites observations 
can provide constant coverage over much of Earth and their images can be merged into a 
single, two-dimensional mosaic depicting, for example, cloud top height for a given area. 
However, this does not provide the three-dimensional (cloud thickness) analysis of clouds 
necessary to meet operational requirements for creating accurate cloud forecasts. 

Currently, the Air Force Weather Agency (AFWA) is using the Cloud Depiction 
Forecast System (CDFS) II to merge satellite imagery and conventional (surface and 
upper air soundings) observations into a global cloud analysis product called Worldwide 
Merged Cloud Analysis (WWMCA). WWMCA initiates the cloud forecast models that 
provides cloud forecasts to the national intelligence community for overhead imagery 
collection planning. To date, there has not been an extensive study to verify operational 
CDFS II products against independent observations (the studies that have been done are 
described later in this chapter). The WWMCA component of CDFS II is difficult to 
verify because there are few independent instruments designed to detect clouds that have 
global coverage that are not used in developing WWMCA products (i.e., few satellite 
observations that are not ingested into CDFS II). 
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Our goal in this study was to begin the testing of CDFS II by verifying the 
operational cloud analysis products generated by WWMCA. Our main research 
questions were: 

1. How should WWMCA products be verified? 

2. How well does WWMCA perform according to the type of cloud condition, or 
cloud event? 

3. How well does WWMCA perform according to the time of day and the time 
of year? 

4. How well does WWMCA perform according to geographic location or 
region? 

B. SCOPE OF THIS RESEARCH 

The purpose of this research was to develop, test, and apply a method to verify 
WWMCA products against cloud information from a space-based radar system, Cloudsat. 
Additionally, we wanted to develop initial verification results for use in evaluating the 
impacts of CDFS II products on the operational planning and operational outcomes of the 
IC users of CDFS II products—that is, to develop information that can be used to answer 
the questions “How do cloud analysis and forecast products affect satellite imagery 
collection? Figure 1 provides an overview of the approach being used in research at the 
Naval Postgraduate School (NPS) to address these operational impacts. The upper left 
portion of this figure summarizes the approach used in this study—comparing cloud 
analyses to cloud observations to verify the analyses. In future studies, we will compare 
cloud forecasts to cloud observations to verify the forecasts. The upper right portion of 
Figure 1 summarizes the approach being used in a companion study at NPS that is 
verifying the impacts of overhead imagery mission planning on mission outcomes 
through the use of a mission planning and outcomes model. The bottom portion of 
Figure 1 indicates how the results from the two types of verification studies will be 
merged to quantitatively assess the impacts of cloud analyses, cloud forecasts, and actual 
clouds on operational planning and operational outcomes. 
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Analyzing Cloud Analysis and Forecast Impacts on Imagery Collection 


Verification of Cloud Analyses and 


Forecasts 


Verification of Operational Plans 



Metrics of the Impacts of Cloud 
Analyses, Cloud Forecasts, and 
Actual Clouds on Operational 
Performance 


Figure 1. A schematic illustration of the approach being used at NPS to quantitatively 
assess the impacts of cloud analyses, cloud forecasts, and actual clouds on 
operational planning and operational outcomes. The upper left portion of the 
figure summarizes the approach being used in this study to verify WWMCA 
cloud analyses. The upper right portion of the figure summarizes the 
approach being used in a companion NPS study to verify operational 
planning. The bottom portion of the figure represents the merger of 
information from the two verification studies. 


C. PRIOR RESEARCH 

The most relevant prior studies for our research were conducted by the University 
Corporation for Atmospheric Research (UCAR), Atmospheric Environmental Research, 
Inc. (AER), and the U.S. Air Force 16 th Weather Squadron. The first was a WWMCA 
verification study in which WWMCA output was compared against Cloudsat data 
(UCAR 2008), and the second was a small follow up effort performed in 2010 (AER 
2010). Details of these studies are available to the United States government personnel 
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and contractors but were never captured in a technical report or paper and are only briefly 
summarized in a verification and validation paper by Northrop Grumman Information 
Systems (NGIS 2011). The studies covered two periods spanning approximately three 
months: 01 April—29 June 2008 (Northern Hemisphere only) and 28 March—31 May 
2010 (Northern and Southern Hemispheres). These studies concluded that WWMCA 
detected clouds when Cloudsat said there was cloud 64% of the time in the Northern 
Hemisphere and more than 80% of the time in the Southern Hemisphere. NGIS (2011) 
reported that WWMCA tended to under analyze cloud, as opposed to falsely assigning 
cloudy conditions to no-clouds conditions. 

The study by the 16 th Weather Squadron (Selin 2011, personal communication), 
was a continuation of the UCAR study but used Cloudsat data to assess WWMCA 
performance in identifying clouds at low, middle, and high levels in the Northern 
Hemisphere. Two three month periods were evaluated: 01 April—29 June 2008 and 
28 March—31 May 2010. The study concluded that WWMCA does an adequate job of 
classifying low and middle clouds correctly, if it is able to detect them. WWMCA had 
the hardest time detecting low and middle clouds. Selin suggests that problems in the 
detection of low and middle level clouds is probably the biggest obstacle when it comes 
to using WWMCA for cloud level and cloud profile information. 

Ruggiero (2000) describes a study conducted by the Air Force Research 
Laboratory (AFRL) to examine processes for using cloud analysis products to initialize 
numerical weather prediction (NWP) models. The study focused on eastern 
Massachusetts in September 1995 and used surface observations and rawinsonde data in 
the verification process. The study concluded that the cloud detection algorithms used in 
WWMCA at the time of the study were able to detect clouds when clouds were present 
81% of the time and agreed with the observed cloud fraction 73% of the time (NGIS 
2011 ). 

Horsman (2007) verified WWMCA against real-time surface weather 
observations at ten different Air Force bases within the continental United States for 
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16 days. This study found that WWMCA performance varied by climate region. 
Overall, the study found that overall, WWMCA performed poorly with a “verification of 
27% and a miss rate of 32%” (Horsman 2007). 

Norquist (2007) verified WWMCA against field measurements collected from 
cloud profiling radar (CPR) and a portable lidar during two efforts at Hanscom Air Force 
Base, MA. From 36 days of data, a total of 117 hours were selected in which both 
instrumental observations and WWMCA were available (by design, cloud cover was 
present in all observations). “Of these 117 hours, WWMCA detected clouds in 91 giving 
WWMCA an overall cloud detection rate of 78%” (NGIS 2011). 

Gustafson et al. (2011) compared Moderate Resolution Imaging 
Spectroradiometer (MODIS) derived cloud mask data from Aqua and Terra to WWMCA 
cloud cover data for June and September of 2010. Cloud fractions for both MODIS cloud 
masks and WWMCA were calculated and classified as either clear (<20%), partly-cloud 
(20—80%), and cloudy (>80%). Table 1 displays the comparison results from Terra for 
June 2010 (Aqua results were very similar). Overall, the cloud fractions of WWMCA 
and MODIS were in agreement 65% of the time. WWMCA performed less well in polar 
regions and relatively well in oceanic glint regions and over bright backgrounds, such as 
deserts. Gustafson concluded that MODIS cloud mask data is less than ideal as an 
independent source of cloud truth. 

Table 1. The results of cloud fraction binning from WWMCA/MODIS (Terra) 
comparison from June 2010. Table based on NGIS (2011). 


Cloud Fractions 

Percentage (%) 

Clear matched 

25 

Partly cloudy matched 

6 

Cloudy matched 

34 


Bartlett (2009) evaluated several methods for verifying WWMCA cloud 
distributions. These methods include comparisons with MODIS imagery and cloud- 
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related information from the Navy Operational Global Atmospheric Prediction System 
(NOGAPS). The study concluded that quantitative verification of cloud distributions can 
be achieved and that more reference distributions based on satellite imagery be 
developed. 

In 2004 and 2011, the 14 th Weather Squadron and AER, looked at the long-term 
mean behavior of WWMCA (NGIS 2011). Their study concluded that the merging 
processes used to develop WWMCA products can introduce errors that are artifacts of the 
merging processes. Most of the artifacts were found at the higher latitudes where 
observations are provided primarily by polar orbiting satellites (NGIS 2011). The study 
also concluded that some of WWMCA’s performance problems arise from physical 
inconsistencies in the merging processes. 

It is difficult to summarize the results of these prior studies, since: (a) they are 
based on different periods, locations, and analysis methods; and (b) most are not well 
documented. However, these studies seem to indicate that: (1) the WWMCA probability 
of detection (POD) for clouds may be, overall, in the range of 65-85%; (2) WWMCA 
seems to under analyze clouds, especially highly optically thin cirrus (in part by design) 
and low clouds over oceans; and (3) verification of WWMCA is problematic due to the 
relative lack of independent and comprehensive verifying data. These prior studies 
assessed WWMCA products from a wide range of years. However, CDFS II has 
undergone many changes since it was first implemented in 2002. There have been 
multiple science level changes and the tunable parameters have been continuously altered 
to improve the baseline performance. Thus, the results from the prior studies are useful 
but may be out of date, or soon may become out of date. 

The prior studies have provided useful results, but they have a number of 
important shortcomings that indicate further studies are needed. These shortcomings 
include: 

1. The data sets, methods, and results for these studies have not been well 
documented. 

2. The studies covered relatively short periods. 
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3. Most of the studies only looked at small regions. 

4. The studies that investigated large regions provided their results at 
relatively coarse spatial resolutions. 

5. The studies suffered from a lack of comprehensive, independent 
observational data against which to verify WWMCA. 

6. The studies apparently used and/or reported only limited verification 
metrics (e.g., POD was reported but not FAR, bias, Heidke skill score, or 
other metrics needed to develop a full verification). 

7. The studies provided little information on how WWMCA performance has 
changed over time, as upgrades to CDFS II have been made. 

To address these shortcomings we designed our study to achieve the following 
objectives. 

1. Design and develop a method to do routine, near real-time verification of 
WWMCA based on Cloudsat observations. 

2. Test the method for a range of cloud events, locations, and periods. 

3. Develop initial verification results using a range of metrics. 

4. Develop initial verification results for use in evaluating the impacts of 
cloud analyses, cloud forecasts, and actual clouds on the planning and 
outcomes of satellite imagery collection operations. That is, provide 
initial information for answering the question: “How do cloud analysis and 
cloud forecast performance, and actual clouds themselves, affect the 
planning and outcomes of satellite imagery collection operations?” 

D. ORGANIZATION OF THESIS 

Chapter II gives background information on CDFS II, WWMCA, and Cloudsat, 
and also discusses the methodology used to verify and compare WWMCA data using 
Cloudsat. Chapter III discusses the results of the WWMCA verification. Chapter IV 
provides a study summary, conclusions, and recommendations for future research. 
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II. DATA AND METHODS 


A. OVERVIEW 

This chapter provides background information on CDFS II, WWMCA, Cloudsat, 
and Cloudsat’s data product 2B-GeoProf. The datasets used for this study and the 
methodology for processing and matching WWMCA and Cloudsat data are discussed. 

B. CDFS II AND WWMCA DESCRIPTION 

1. Cloud Depiction Forecast System (CDFS) II 

The Cloud Depiction Forecast System (CDFS) II is a computer processing system 
designed and operated by the Air Force Weather Agency (AFWA) that produces an 
hourly, global, cloud analysis from individual satellites and merges them with 
conventional observations (e.g., surface observations and upper air soundings) into a 
Worldwide Merged Cloud Analysis (WWMCA) that reports the following parameters 
(HQ AFWA/DNXM 2011): 

• total cloud percentage 

• layered cloud percentage 

• layered cloud type 

• layered cloud top heights 

• layered cloud base heights 

• pixel mean time in Julian minutes 

Figure 2 illustrates the different CDFS II components, including WWMCA, how they are 
related to each other, and the process for generating cloud analyses and forecasts. Both 
polar orbiting and geostationary satellites provide observations to CDFS II. The polar- 
orbiters are those in the Defense Meteorological Satellite Program (DMSP) and National 
Oceanic and Atmospheric Administration (NOAA) Polar Orbiting Environmental 
Satellites (POES) program. The geostationary satellites include those in the 
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Geostationary Operational Environmental Satellite (GOES) and the European Space 
Agency’s Meteorological Satellite (METEOSAT) programs. The satellite and 
conventional observations undergo a four level process to be merged into a global cloud 
analysis. Level one is data calibration, level two classifies each pixel into cloudy or 
clear, level three applies cloud layering and typing, and level four consists of merging the 
separate analyses into one global analysis, called WWMCA. WWMCA is used to initiate 
a suite of cloud forecast models (HQ AFWA/DNXM 2011). Refer to Appendix A for a 
detailed description of the four processing levels. 



Inputs Analysis 

1 Defense Meteorological Satellite Program 

2 Polar Operational Environment Satellite 

3 Geostationary Operational Environment Satellite 

4 Meteorological Satellites operated by European Space Agency 

5 Worldwide Merge Cloud Analysis 


Output Forecast 

6 Forecaster in the loop 

7 Advect Cloud 

3 Stochastic Cloud Forecast Model (Global) 

9 Diagnostic Cloud Forecast (Regional) 


Figure 2. Schematic overview of the Cloud Depiction Forecast System II. 

Observational data are received from meteorological satellites, conventional 
observations, and global analyses from various models. These data are 
merged into one global cloud analysis, called WWMCA, which is used to 
initiate cloud forecast models. Quality control is performed on the cloud 
analysis and model output products by a forecaster. Figure from HQ 

AFWA/DNXM (2011). 


WWMCA is known to have some deficiencies (HQ AFWA/DNXM 2011). First, 
WWMCA often places high cirrus clouds at too low an altitude due to surface 
radiation emission contamination in the brightness temperatures. Second, CDFS II tends 
to under analyze low clouds. A correction is required because low cloud top 
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temperatures may be very close or exceed that of the underlying surface temperatures, 
thus producing a false colored pixel (HQ AFWA/DNXM 2011). 

2. Worldwide Merge Cloud Analysis (WWMCA) 

WWMCA provides information for a global grid of square cells that each covers 
an area of 24 km by 24 km (true at 60° latitude). For each cell, the following information 
is provided: 

• i and j spatial coordinates 

• total cloud percentage 

• layered cloud percentage 

• layered cloud type 

• layered cloud top heights 

• layered cloud base heights 

• mean pixel time in Julian minutes 

Figure 3 shows the data format of a WWMCA file. The total cloud amount is the 
percentage of cloud-filled pixels divided by the by the total number of pixels per cell. 
Layered cloud amount uses the same calculation as total cloud amount, except that the 
calculation uses the coldest (top) layer pixels. Cloud base height is calculated using the 
RTNEPH (Real Time Nephanalysis) technique (Kiess and Cox 1988). Cloud top height 
is calculated by interpolation of the Naval Operational Global Atmosphere Prediction 
System (NOGAPS) atmospheric temperature and height profiles to the mean cloud top 
temperatures for each grid cell. The highest percentage of cloud type in the final analysis 
decides cloud type for the cell (HQ AFWA/DNXM 2011). WWMCA is used to 
initialize three cloud forecast models: 

1. Advect Cloud (ADVCLD) 

2. Stochastic Cloud Forecast (SCFM) 

3. Diagnostic Cloud Forecast (DCF) 
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WWMCA File Format 


Year 

Month 

Day 

Hour 


20040 

20040 

20040 

20040 

20040 

20040 

20040 

20040 

20040 


] 03 000773 1 
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] 03 000776' 
] 03 000777' 
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50272 

704677 
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00000- 
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06477 
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Layer 2 


oooooooooc 

4|0 6 67 4|00 0|C 00000000OC 
OOOOOCOOOCOOOOOOOOOCOOO)di 


OOOOOOOOOC' 

0329304493 

OOOOOOOOOC' 

OOOOOOOOOC' 

OOOOOOOOOC' 

OOOOOOOOOC' 


0000< 

0000< 

0000( 

0000 ( 

0000( 

0000( 

0000 < 

0000 ( 

0000 ( 


Hemisphere 


Within each laver: 

Percent coverage at that layer 
Cloud type (1-9) 

Height of base (meters) 
Height of top (meters) 


>000(999100 
>000(999100 
>000( 999100 
>000(999100 
>000( 999100 
>000(999100 
>000( 999100 
>000( 999100 
>000(999100 


Total cloud amount 
from all layers 
(percent) 


Cloud Type Table 

1 

Cb 

4 

Cu 

7 

Ac 

2 

St 

5 

As 

8 

Cs 

3 

Sc 

6 

Ns 

9 

Ci 


Figure 3. Worldwide Merge Cloud Analysis file format. Figure from HQ 

AFWA/DNXM (2011). 


C. CLOUDSAT 

1. Description of Cloudsat 

Cloudsat is a National Aeronautics and Space Administration (NASA) operated 
satellite launched on April 28, 2006. It carries a W-band (94 GHz) cloud profiling radar 
(CPR) for sensing condensed cloud particles and detecting precipitation, and is part of the 
Earth Systems Science Pathfinder (ESSP) mission. The original Cloudsat program was 
funded for 22 months and has been extended to September 2011. Ground operations are 
performed by the United States Air Force (USAF) at Kirkland Air Force Base in 
Albuquerque, New Mexico. The CPR has been providing continuous, global time series 
of vertical cloud structure and properties at a vertical resolution of 485 m since June 2, 
2006 (Stephens et al. 2008). 

Cloudsat was designed to help fulfill four mission objectives: “(1) quantify the 
representation of clouds and cloud processes in global atmospheric circulation models, 
(2) quantify the relationships between vertical profiles of cloud liquid water and ice and 
radiative heating of the atmosphere and surface; (3) evaluate cloud properties retrieved 
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from other satellites system (e.g., Aqua); and (4) contribute to improving the 
understanding of the indirect effect of aerosols on clouds by investigating the effect of 
aerosols on cloud and precipitation formation” (Stephens et al. 2008), which is critical to 
the understanding of climate change. 

The Cloudsat satellite flies in a constellation of satellites referred to as an “A- 
Train,” which consists of the Cloud-Aerosol Lidar and Infrared Pathfinder Satellite 
Observation (CALIPSO) satellite, followed by other NASA satellites (Figure 4). 
CALIPSO has a two-wavelength, polarization-sensitive backscattering lidar that provides 
high resolution vertical profiles of clouds (NASA Facts-Cloudsat 2003). Cloudsat uses 
near-nadir pointing millimeter-wavelength radar capable of probing the vertical structure 
of a cloud (Cloudsat Data Products Handbook 2011). The combination of data from 
theses satellites provides a valuable source of cloud information. The other satellites that 
makeup the “A-Train” are Aura, Polarization and Anisotropy of Reflectances for 
Atmospheric Sciences coupled with Observations from a Lidar (PARASOL), Aqua, and 
Orbiting Carbon Observatory (OCO). 



Figure 4. 


Depiction of the satellites that make up the “A-train”. Each satellite 
has its name and time of equator crossing listed. Note that CALIPSO trails 
Cloudsat by 15 seconds to allow for synergy between Aqua, Cloudsat, and 
CALIPSO. From NASA Facts-A-Train 2003. 
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The constellation flies in a Sun-synchronous orbit with a mean equatorial altitude 
of 705-730 km and an inclination of 98.2°. This orbit is fixed, so that there are no 
changes in the orbital elements over long time periods. The revisit time is 16 days. This 
means that Cloudsat repeats ground track every 16 days, or 233 revolutions (Cloudsat— 
Education: FAQ 2011). 

Clouds are weak scatters of microwave radiation, in contrast to the stronger 
reflections from Earth’s surface. To detect these weak cloud signals, the CPR was 
designed for a minimum detectable signal, Z min . of -28 dBZ (Stephens et al. 2008). Table 
2 summarizes the defining parameters of the radar confirmed by measurements before 
and after launch. The CPR emits a 3.3 microsecond pulse resulting in a vertical 
resolution of 485 m. The back scattered signal is oversampled to produce a range gate 
spacing of 240 m. From the altitude of 705 km, the instantaneous field of view (IFOV), 
at mean sea level, is 1.7 km along and 1.3 km across track (Figure 5), and 688 pulses are 
averaged to produce a nominal footprint of 2.5 km along track. The volume defined by 
the along track footprint and 240 m range is referred to as a range resolution volume 
(RRV) (Mace et al. 2007). 


Table 2. Cloud profiling radar (CPR) parameters and their respective 

performances. From Stephens et al. 2008. 


Parameter 

Performance 

Frequency 

94.05 GHz 

Altitude 

705-730 km 

Range resolution (6 dB) 

485 m 

Cross-track resolution 

1.4 km 

Along-track resolution 

1.8 km 

Pulse width 

3.3 (js 

Peak power (measured) 

32.6 dB 

PRF 

3700-4300- Hz 

Antenna diameter 

1.85 m 

Antenna gain 

63.1 dBi 

Antenna side lobes 

-50 dB @ 0 > 7° 

Integration time (single-beam) 

0.16 s 

Data window 

30 km 

Minimum detected reflectivity (measured) 

-30 dBZ 
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Figure 5. The instantaneous field of view (IFOV) for the Cloudsat cloud profiling radar 

(CPR). From Cloudsat Product Handbook (2011). 

Due to the radiometric differences between cold scenes (e.g., clear skies over a 
cold surface at night) and hot scenes (e.g., clear skies over a warm surface during 
daytime), the single-beam Z m j n varies by ± 1 dB. Cloudsat’s calibration as of 2008 of 
Z min is 29.9 to 30.9 dBZ (Stephens et al. 2008). 

Cloudsat has problems detecting some low level strata, cumulus, non-drizzling 
stratocumulus, warm altocumulus composed of small water droplets, and optically thin, 
high cirrus (Mace et al. 2007). Since the CPR vertical resolution is 485 m and surface 
reflectivity is up to five times greater than cloud reflectivity, measurements in the lowest 
two RRVs of the profile are dominated by surface back scatter. This surface 
contamination can extend up to 1 km, and the signal usually returns to maximum 
sensitivity by the 5 th RRV. However, on average, clouds that reflect above 10 dBZ in the 
3 ld and 4 th RRVs (i.e., 480 to 960 m above surface) are detected above surface 
contamination (Mace et al. 2007). Sassen and Khvorostyanov (2007) explain how the 
radiative and backscattering properties of mixed-phase clouds effect their detection. 

2. Description of 2B Cloud Geometrical Profiling (GeoProf) Product 

The 2B Cloud Geometrical Profiling Product (GeoProf) identifies the levels in the 
vertical column sampled by Cloudsat that contain significant radar echo from 
hydrometeors and provides an estimate of the radar reflectivity factor for each of these 
volumes. Details on the Geoprof algorithms are provided in the Level 2 GEOPROF 
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Product Process Description and Interface Control Document (2007). The 2B GeoProf 
product uses an algorithm that reads the Cloudsat 1B-CPR granule, which measures the 
backscatter power as a function of distance from the radar, and the corresponding 
MODIS-AUX and ECMWF-AUX granules to produce the cloud mask (Cloudsat 
Standard Data Products Handbook 2008). MODIS’ cloud fraction from the visible 250 m 
MOD35 product is integrated over the Cloudsat footprint (CloudSat 2B GEOPROF 
Quality 2007). 

The cloud mask data is stored in the 2B-GeoProf data product and contains a 
value between 0 and 40 for each range bin, with values greater than 5 indicating the 
likelihood of hydrometeors (Stephens et al. 2008). Larger values indicate a higher 
likelihood of hydrometeors and a lower likelihood of false detections (Table 3). A 
Cloudsat data file contains one granule, where a granule is the data collected during one 
orbit. Granules are broken into 31 segments that allow for a visual view of the data 
(Figure 6). 


Table 3. Description of Cloudsat cloud mask values, false detection rates, and false 
detections. The percentage of false detection is 100 times the number of 
false detections divided by the total number of detections for the specific 
cloud mask value. From Stephens et al. (2008). 


Mask Value 

Meaning 

% false detections goal 

Estimated % false detection 
via CALIIPSO comparison 

-9 

5 

6-10 

Bad or missing radar data 

Significant return power but likely surface clutter 
Very weak echo (detected using along-track 

<50% 

44% 

20 

averaging) 

Weak echo (detection may be artifact of spatial 

<16% 

5% 

30 

correlation) 

Good echo 

<21% 

4.3% 

40 

Strong echo 

<0.2% 

0.6% 
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Figure 6. A visualization of Cloudsat data for one segment of a granule. The top pane is 
the 1A Aux data and contains the date and start time of the granule, and the 
starting/ending latitude-longitude and times of that segment. The middle panel 
is the 2B-GeoProf cloud masking, which is color coded for masking values. 
The bottom panel is an Aqua MODIS image (11 pm) of the granule, which 
was taken one minute prior to the data collection of the granule. From 
Cloudsat Data Processing Center (2011). 


D. STUDY REGIONS AND PERIODS 

We selected two regions for this study based on open source information about 
where the U.S. national intelligence community has a high interest in electro-optical (EO) 
imagery. We selected WWMCA Box 22 (Figure 7) because it encompasses much or all 
of Iraq, Iran, Afghanistan, and Pakistan, all of which are important in U.S. national 
security. WWMCA Box 22 contains mountains, deserts, ocean, and coastal regions and 
mostly resides in the mid-latitudes, extending south to 18°N and north to 50° N, and 
within the field of view (FOV) for geostationary satellites. The diversity of climates, 
terrains, and the availability of geostationary satellite information for use in developing 
WMCA products, makes Box 22 a good case for testing the mid-latitude performance of 
WWMCA. We also selected WWMCA Box 29 (Figure 7) because it too covers areas of 
interest to the IC. Box 29 encompasses much of western Russian and the Barents Sea in 
the Arctic, a region of increasing interest due to decreases in Arctic sea ice and the 
potential for increased Arctic maritime activity. Box 29 extends between 50° N and 
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90° N, so WWMCA products for this box rely on polar orbiting satellites. Thus, Box 29 
is a good case for assessing the high latitude performance of WWMCA. 

The period of the study was January, April, July, and October 2010. These 
months were chose to represent the four main seasons of the year. We limited our study 
to just two WWMCA boxes, and four months of one year due to time constraints. 
However, the methods developed in our study can be readily applied to assess the 
performance of many other boxes, months, and years. They can also be readily applied to 
look at regions smaller or larger than one WWMCA box, and to periods less than or 
greater than one month. See Chapter IV for more information on applications of methods 
and extensions of our study. 



Figure 7. Location of WWMCA boxes 22 and 29, shown by the white southern and 
northern boxes, respectively. These boxes appear skewed in the image due to 
the map projection. Box 22 includes much or all of Iraq, Iran, Afghanistan, 
and Pakistan. Box 29 includes much of western Russia and the Barents Sea. 
Base image from Google Earth (2011). 
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E. 


DATA SETS 


1. WWMCA 

The WWMCA data set used for this study was provided by the 16 th Weather 
Squadron and consisted of global, hourly cloud analyses for all of 2010 in an ASCII 
format. The file name convention is: the prefix CLD, year, month, and WWMCA box 
number. For example, the data files for WWMCA Box 22 for March and October 2010 
are CLD201003022 and CLD201010022, respectively. The data files contain all the 
hourly cloud analyses for a WWMCA box for an entire month and have about 1.5 million 
lines of data per file. One zipped WWMCA data file is about 100 MB and has an 
unzipped-to-zip ration of 10:1. 

2. Cloudsat-2B-Geoprof 

The Cloudsat 2B-GeoProf data was downloaded from the Cloudsat Data 
Processing Center and are in a Hierarchical Data Format (HDF). Each file has 37,100 
lines of data and the name convention is: the Julian date and granule start time (hour, 
minute, second), granule number, type of sensor (Cloudsat = CS), data product type, and 
the suffix ‘GRANULE_P_R04E03.hdf.’ For example, granule 19893 is named 
2010023092101 _19893_CS_2B-GEOPROF _GRANULE_ P_R04_E03.hdf. One 
zipped Cloudsat data file is about 13 MB and has an unzipped-to-zip 3:1. 

F. METHODS 

The Cloudsat and WWMCA data sets are on different temporal and spatial grids 
that have different resolutions, and differ in the variables they use to describe clouds. So 
the data needed to be processed and put on a common footing before it could be 
analyzed. This section describes the programming requirements and associated tasks 
necessary to compare and analyze the two sets of data in a spreadsheet program (e.g., 
Microsoft Excel). The major challenge was getting the Cloudsat data: (a) onto the same 
time and space grid as the WWMCA data; and (b) into the same units as the WWMCA 
data. This consisted of matching Cloudsat data to the WWMCA cells, converting 

Cloudsat cloud mask measurements into total cloud amount percentages for a given 
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WWMCA cell, and binning both WWMCA and Cloudsat data into common cloud cover 
events. The WWMCA variable of interest for this research project was total cloud 
amount. 

The WWMCA data is organized into 128 square boxes (64 boxes per hemisphere) 
with each box containing 128 cells (Figure 8). The cells have unique i-j coordinates that 
describe the latitude and longitude of the cell. The cells are 24 x 24 km, true at 60° 
latitude (HQ AFWA/DNXM 2011). 


WWMCA Box and Grid Organization 


Northern Hemisphere 



Boxes 1-8 
Boxes 9-16 


Boxes 57-154 


Southern Hemisphere 



Boxes 101-103 
Boxes 109-116 


Boxes 157-164 


Figure 8. The WWMCA grid is organized into 128 square boxes (64 boxes per 
hemisphere) with each box containing 128 cells. The cells have unique i-j 
coordinates that describe the latitude and longitude of the cell. The cells are 
24 km long on a side (true at 60° latitude). The yellow shaded boxes are the 
two WWMCA boxes that we focused on in this study. The northern box is 
WWMCA Box 29 and the southern one is Box 22. Base figures from 16 th 

Weather Squadron (2011). 


Each Cloudsat granule is 40,786 km in length and contains 37,088 profiles. Each 

profile represents a vertical sounding from the satellite through the atmosphere to the 

surface. The horizontal surface area represented by each profile is the Cloudsat pixel 

area (or IFOV). Each Cloudsat pixel (Figure 5) is about 1.3 km wide (across the track) 

and 1.7 km long (along the track). The main Cloudsat data of interest for this study was 

the Cloudsat cloud mask data in the 2B-Geoprof product, which provided information 
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about the presence or absence of clouds, along with corresponding date-time and latitude- 
longitude data. This data was available for each profile and for each of 125 vertical 
levels within the profile (Figure 9). The cloud mask values indicate the likelihood of 
cloud detection and are described in Table 3. The Cloudsat dataset also includes a 
variable that describes the height of the Cloudsat levels above mean sea level (MSL) that 
we used to filter out the backscatter from the surface (see next section). 


Need 1 representative 
value for cloud mask. 
This representative 
value will be the 
highest cloud mask 
value in the column. 


Cloudsat (altitude of 705 km) 


Cloud Profiling Radar (CPR) Beam 


Top of Atmosphere 


Surface 


Column 1 


Vertical CPR levels 

(240 meter resolution, 

124 levels contained in 
CPRCIoudmask data) 

The columns listed in the "CPR_Cloud_mask" 
parameter of the granule HDF file represent 
the vertical levels of the CPR starting in the 
upper atmosphere (column 1) and ending at 
the bottom of the atmosphere (column 124). 


Column 124 


Figure 9. Vertical levels in Cloudsat. 
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Figure 10 illustrates the processing of the WWMCA and Cloudsat data that we 
did to prepare the data for analysis, as well as the analysis of the processed data. 



Figure 10. Overview of data processing conducted to prepare WWMCA and Cloudsat 

datasets for analysis. 

1. Cloudsat Data Preparation 

Cloudsat data was prepared for verifying the WWMCA data by running a set of 
Python programs that extracted the required data from a data base containing the 2010 
Cloudsat data files that intersected WWMCA Boxes 22 and 29. The required data were: 

• profile time 

• UTC start time 

• latitude 

• longitude 

• height 

• CPR cloud masking 
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The data for these variables were extracted and matched to the appropriate WWMCA 
verification time and cell. The Cloudsat observations were matched to WWMCA 
verification times that best represented the atmospheric cloud state for that time, using a 
similar to that used by CDFS II to match satellite observations to the cloud analysis. 
Cloudsat observations that fall within the first (second) half of an hour were matched to 
WWMCA data of that (the following) hour. For example, if a Cloudsat observation was 
from 0817Z, it was matched to the 0800 WWMCA verification time. If the observation 
was from 0831Z, it was matched to the 0900 verification time. Next, the Cloudsat data 
was mapped, or converted, to the WWMCA grid and matched to the corresponding 
WWMCA cells within the WWMCA box of interest, either Box 22 or Box 29. 

We used the height variable to remove from the data processing effort any 
Cloudsat observations with heights less than 1 km, to eliminate backscatter from the 
ground. We also identified the WWMCA cells that contained six or more Cloudsat 
pixels, so that we could focus our analyses on the cells and times for which there was at 
least 25% coverage by Cloudsat (six Cloudsat pixels cover 25% coverage of a 
WWMCA). 

Cloudsat cloud masking data are available at 125 vertical levels for each profile 
(Figure 9). For comparison to the WWMCA total cloud amount data, we reduced the 125 
values to a single representative cloud mask value for each profile. We used the highest 
cloud mask value from the 125 levels as the representative cloud mask value (after 
deleting the values from the lowest 1 km, as described above). There are pros and cons 
to this selection of the cloud mask value. In particular, this selection minimizes the risk 
of overstating the cloud masking, but it may underestimate the actual cloud masking. 
Recall that the cloud masking values represent the likelihood of occurrence of 
hydrometeors in a profile. So a summation, or an average, of values from more than one 
of the 125 levels might well overstate the likelihood clouds. Thus, we chose to use the 
highest cloud mask value to represent the full profile. 
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We assigned a cloud occurrence value to each pixel by using the Cloudsat cloud 
mask value for each Cloudsat pixel to determine if cloud or no-cloud conditions had 
occurred. If the cloud mask value was less than 30, we assigned a cloud occurrence value 
of 0 to indicate a no-cloud condition. If the cloud mask value was 30 or greater, we 
assigned a cloud occurrence value of 1 to indicate the presence of cloud, or cloud 
conditions. It is important to note that prior studies used a Cloudsat cloud mask value of 
20 as the cloud/no-cloud threshold, rather than the value of 30 that we used. The 
Cloudsat Standard Data Products Handbook (2008) states that values of 30 or greater are 
associated with the greatest confidence in cloud detection. Based on this, we decided to 
use a cloud mask value of 30 as the threshold for determining whether clouds were or 
were not present in a profile. However; to test the sensitivity of our analysis results to 
this threshold selection, we also used a threshold value of. 

Next, we converted the set of Cloudsat cloud and no-cloud condition values for all 
the pixels in a WWMCA cell to a single total cloud amount for the cell using the 
following calculation: 

Number o f Cloudsat cloudy points ... , „ 

—-;-:- r—. -;-;- * 100 = Total cloud amount (%) 

Total number of Cloudsat points 

That is, we divided the number of Cloudsat pixels that indicated the presence of clouds 
within a WWMCA cell by the total number of Cloudsat pixels in the cell, and then 
multiplied by 100, to derive a total cloud amount percentage for the cell. For example, if 
12 out of 19 Cloudsat pixels in a WWMCA cell showed cloud conditions, then the total 
cloud amount for the cell was calculated to be 63% (Figure 11). 


24 




12 of 19 points are cloud. The total 
cloud amount is 63% 


Figure 11. Schematic illustration of the overlap of a Cloudsat swath (red line) and a 
WWMCA cell (gray box). The red circles indicate the Cloudsat 
representation of cloud conditions (solid red circles) and no-cloud conditions 
(open red circles). The total cloud amount for the cell, as indicated by 
Cloudsat, is the number of cloud conditions divided by the number of cloud 
and no-cloud conditions, all multiplied by 100. For this case, 12 of the 19 
circles indicate cloud conditions, so the total cloud amount is 63%. 


The resulting Cloudsat total cloud amount data was then binned into one of three cloud 
cover events (Table 4). Cells that had a total cloud amount value > 80% were assigned to 
the Definite-Cloud event bin. Cells that had a total cloud amount value < 20% were 
assigned to the No-Cloud event bin. Cells with in between total cloud amount values 
were assigned to the Probable-Cloud event bin. 


Table 4. Method used to determine cloud cover event status for a WWMCA cell based 

on the Cloudsat-based total cloud amount (%) for the cell. 


Cloud Event Number 

Total Cloud Amount 

Event Name 

1 

< 20 % 

No-Cloud 

2 

21—79 % 

Probable-Cloud 

3 

> 80 % 

Definite-Cloud 


2. WWMCA Data Preparation 

We needed to match up the WWMCA total cloud amount data to the Cloudsat 
data in the CSV file that was generated during the Cloudsat data preparation stage by 
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inserting WWMCA data in the appropriate rows. The rows represent the time and 
location of the data. So inserting the WWMCA data into the correct rows meant that we 
did a temporal and spatial match of the Cloudsat and WWMCA data. 

We extracted WWMCA data and wrote it to a CSV file for the period and 
location of the Cloudsat data that was already in the CSV file. However, since the 
WWMCA data has a lower temporal and spatial resolution than the Cloudsat data, there 
are rows that contain Cloudsat data and duplicate WWMCA data from adjacent rows. 
All of the WWMCA information contained in a data line (Figure 3), including the mean 
pixel age was written to the CSV file containing the Cloudsat data. 

We then binned WWMCA total cloud amount into the same three cloud cover 
events as Cloudsat total cloud amount (Table 4). 

In summary, there are significant differences in the spatial and temporal 
resolutions of WWMCA and Cloudsat data. WWMCA data is contained in 
1024 x 1024 cells per hemisphere with a horizontal resolution of 24 km. Cloudsat data 
has a 1.3 x 1.7 km footprint and is referenced to latitude and longitudes. CDFS II outputs 
a new WWMCA product every hour, and Cloudsat flies over a WWMCA cell for three 
minutes, twice daily. This indicates three significant challenges in verifying WWMCA 
against Cloudsat (and many other observational data sets): (1) accurately matching the 
two data sets in space; (2) accurately matching the two data sets in time; and 
(3) obtaining a sufficient amount of independent observational data to use in verifying 
WWMCA (finding an adequate amount of independent observations that fall within a 
WWMCA cell to confidently verify the WWMCA analysis for that cell). Cloudsat and 
many other global observational data sets are readily available on regular latitude- 
longitude grids. Verification of WWMCA would be a much more straightforward 
process if WWMCA data was also readily available on a regular latitude-longitude grid. 

3. Analyzing Data in Microsoft Excel 

Verification of the WWMCA cloud analysis to Cloudsat data was accomplished 
with Microsoft Excel. Refer to Appendix B for the format and formulas used to analyze 
the data. The CSV file containing the processed WWMCA and Cloudsat data for a given 
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month was imported into Excel and data lines with values of “-1” and “less than 6 points” 
were filtered out of the analysis process, since these points represented missing data and 
WWMCA cells with less 25% area covered by Cloudsat observations, respectively. 

Next we tallied the hits, false alarms, misses, and correction rejections for the 
three cloud cover events for Cloudsat cloud thresholds values 20 and 30. The tallies were 
generated for use in entering data into the contingency tables we used to verify the 
performance of WWMCA. Table 5 explains the logic for determining whether a given 
WWMCA-Cloudsat data pair would be counted as a hit, false alarm, miss, and correction 
rejection. In this table, the upper portion describes the logic for verifying the 
performance of WWMCA in analyzing Definite-Cloud events (cloud event 3), the middle 
portion does the same for Probable-Cloud events (cloud event 2), and the bottom portion 
does the same for No-Cloud events (cloud event 1). The cell letters (A, B, C, D) refer to 
the four cells in the two by two contingency tables that we used (see next section and 
Wilks 2006). For example, the third row of Table 5 shows that if WWMCA analyzed 
cloud event 3 and Cloudsat observed cloud event 3, then a tally would be entered in cell 
A of the contingency table, which is the Hit cell. 


Table 5. Logic for tallying hits, false alarms (False), misses (Miss) and correct 

rejections (Reject) when comparing a WWMCA cloud cover analysis to the 
corresponding Cloudsat cloud cover observation. 


Tallying Logic for Verifying WWMCA Performance in Analyzing Cloud Cover Events 

3 

Cell 

WWMCA 

Cloudsat 

Hit 

A 

3 

3 

False 

B 

3 

lor 2 

Miss 

C 

lor 2 

3 

Reject 

D 

lor 2 

lor 2 

2 

Cell 

WWMCA 

Cloudsat 

Hit 

A 

2 

2 

False 

B 

2 

1 or 3 

Miss 

C 

lor 3 

2 

Reject 

D 

lor 3 

1 or 3 

1 

Cell 

WWMCA 

Cloudsat 

Hit 

A 

1 

1 

False 

B 

1 

2 or 3 

Miss 

C 

2 or 3 

1 

Reject 

D 

2 or 3 

2 or 3 
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Next, we took the tallied categories for cloud cover events and then flagged the 
data as either day or night. This was done by using the monthly mean sunset and sunrise 
times for a reference point within a WWMCA box that we assumed to be representative 
to the entire box (Table 6). The reference point we used for Box 22 was Tehran, Iran. 
The reference point for Box 29 was Orenburg, Russia. In Box 29, data lines that fell 
within the Arctic Circle (Cloudsat latitudes > 70°N) were flagged as either day or night 
depending on the time of year. January and April were assumed to be polar night 
conditions, and July and October were assumed to be polar day conditions. We then 
created separate tallies for the day and night hit, false alarm, miss, and correction 
rejection categories. This allowed us to separately assess the performance of WWMCA 
for all hours of the day, for daylight hours, and for nighttime hours. 
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Table 6. 


Mean times of sunrise and sunsets for reference points within WWMCA Box 22 and Box 29. We assumed that the 
reference point times were representative of the entire WWMCA box. The daylight times were used to flag WWMCA 

and Cloudsat data as either day or night data. 


Box 22 Reference: Tehran, Iran Daylight Range 


Box 22 

Sunrise 

(Local) 

Sunset 

(Local) 

Sunrise 

(Local) 

Sunset 

(Local) 

GMT 

+6 hrs 

GMT 

+6 hrs 

Sunrise 

(Z) 

Sunset 

(Z) 

Sunrise 

(Z) 

Sunset 

(Z) 

Filter Type 

Jan 

9:30 

17:30 

0.3958 

0.7292 

3:30 

0.1458 

0.5417 

0.8750 

13:00 

21:00 

Day 

Apr 

7:30 

21:00 

0.3125 

0.8750 

3:30 

0.1458 

0.4583 

1.0208 

11:00 

0:30 

Night 

Jul 

6:00 

22:30 

0.2500 

0.9375 

3:30 

0.1458 

0.3958 

1.0833 

9:30 

2:00 

Night 

Oct 

8:30 

19:30 

0.3542 

0.8125 

3:30 

0.1458 

0.5000 

0.9583 

12:00 

23:00 

Day 


Box 29 Reference: Orenburg, Russia Daylight Range 


Box 29 

Sunrise 

(Local) 

Sunset 

(Local) 

Sunrise 

(Local) 

Sunset 

(Local) 

GMT 

+6 hrs 

GMT 

+6 hrs 

Sunrise 

(Z) 

Sunset 

(Z) 

Sunrise 

(Z) 

Sunset 

(Z) 

Filter Type 

Jan 

9:00 

18:00 

0.3750 

0.7500 

6:00 

0.2500 

0.6250 

1.0000 

15:00 

0:00 

Day 

Apr 

7:00 

21:00 

0.2917 

0.8750 

6:00 

0.2500 

0.5417 

1.1250 

13:00 

3:00 

Night 

Jul 

6:30 

22:30 

0.2708 

0.9375 

6:00 

0.2500 

0.5208 

1.1875 

12:30 

4:30 

Night 

Oct 

8:30 

19:30 

0.3542 

0.8125 

6:00 

0.2500 

0.6042 

1.0625 

14:30 

1:30 

Night 
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4. 


Contingency Table Analysis 


The performance of WWMCA was verified using 2x2 contingency table 
methods (Table 7; Wilks 2006). Even though WWMCA is an analysis we are treated it 
as an hourly forecast so that we could assess its performance using standard performance 
metrics. We calculated a number of performance metrics based on Table 7. 


Table 7. The format of the 2x2 contingency table analysis used to verify WWMCA 

cloud analyses. Based on Wilks (2006). 



Observations 

(Cloudsat) 


Cloud 

No-cloud 

Analysis 

(WWMCA) 

Cloud 

A 

Hits 

(A/N) 

B 

False Alarms 
(B/N) 

A + B 
(A + B) / N 

Marginal Totals for 
Analysis & 
Marginal 
Distributions for 
Analysis 

No-cloud 

C 

Misses 

(C/N) 

D 

Correct 

Rejections 

(D/N) 

C + D 

(C + D)/N 


A + C 
(A + C) / N 

B + D 
(B + D) / N 

N = A + B + C + D 
1.00 


Marginal Totals for Observations 
& Marginal Distributions for 
Observations 

Sample Size & 
Total Probability 


We calculated a number of WWMCA performance metrics based on Table 7. 
The probability of detection (POD), also called hit rate (H), is the ratio of correct 
analyses of an event to the number of times the event occurred. In our case, the events of 
interest were Definite-Cloud, Probable-Cloud, and No-Cloud events, which we analyzed 
separately. The H or POD was calculated by: 


Hits 

H = POD = --- 

(Hits + Misses ) 


Proportion correct (PC), which Wilks (2006) states is the most straightforward 
and sensitive measure of the accuracy of nonprobabilistic forecasts for discrete events, 
credits correct cloud and no-cloud analyses equally, and is the ratio of the number of 
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correct analyses to the total number of samples. However, PC does not distinguish 
between correct event and no-event analyses. We calculated the proportion of occasions 
when the analysis was correct by: 

(Hits + Correct Rejections') 

Number of Samples 


We also calculated the threat score (TS), or critical success index (CSI), which 
eliminates from consideration correct rejections. The best possible threat score is one and 
the worst is a zero (Wilks 2006) and was calculated by: 

Hits 

TS = CSI = ------- 

(Hits + False Alarms + Misses) 


The bias (B) is the total number of analyses of an event divided the total number 
of observations of the event. Unbiased analyses would have a bias value of 1. Bias 
values greater than one indicate that the event occurred more often than it was observed 
(the event was over analyzed) and bias values less than one indicate the event occurred 
less often than observed (the event was under analyzed). We calculated bias by: 

Total for Analysis (Hits + False Alarms) 

Total for Observations (Hits + Misses) 

The false alarm ratio (FAR) is the ratio of false alarms for an event to the total 
number of analyses of the event, or the fraction of analyses that were wrong (Wilks 
2006). Thus, FAR is the number of false alarms divided by the sum of the hits and false 
alarms, all for the event of interest. The best value for FAR is zero and worst value is 
one. The false alarm ratio was calculated by: 
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False Alarms 


FAR = 


T otal for Analysis 


The skill of the analysis was tested with the Heidke skill score (HSS). The 
Heidke skill score is based on the portion correct as the basic accuracy measure. Perfect 
analyses would yield a score of one; a score of zero would indicate that the analyses are 
equivalent to random reference analyses; and a negative score would indicate that the 
analyses were worse than random analyses. Wilks (2006) explains that the reference 
measure for the Heidke skill score is the portion that is correct from a random analysis 
statistically independent of observations. We calculated the Heidke skill score as: 


HSS 


2{Hits * Correct Rejections — False Alarms * Misses) 

( Hits + Misses){Misses + Correct Rejections ) + (/fits + False Alarms)(False Alarms + Correct Rejections) 


A description of our WWMCA performance results is provided in Chapter IV. 


5. Summary of Assumptions 

In order to verify WWMCA against Cloudsat, we made a number of important 
assumptions. In particular, we assumed that: 


1. Cloudsat is the ‘truth’ for describing the actual clouds in the atmosphere. 

2. The Cloudsat level with the high cloud mask value is representative of all 
125 levels and provides a good measure of the total cloud mask. 

3. Cloudsat observations that cover >25% of a WWMCA cell are adequate to 
represent all the clouds in the cell and for verifying WWMCA for that cell. 

4. The mean monthly sunrise/sunset times near the center latitude of a 
WWMCA box are representative of the whole box. 

5. January, April, July, and October are representative of the major seasons. 

6. Cloud cover can be adequately represented by the three cloud events 
described in Table 4. 
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7 . 


WWMCA performance can be adequately assessed using metrics based on 
2x2 contingency tables. 


We applied these assumptions for a variety of reasons—in particular to keep our 
analyses relatively straightforward. Assumption 1 is not easy to modify, since extensive 
independent cloud observations are scarce. Assumptions 2-7 can be modified and we 
recommend doing so in future studies. In addition, we recommend testing the sensitivity 
of the performance results to these assumptions (see Chapter IV). 
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III. RESULTS 


A. OVERVIEW 

In this chapter, we present the results of WWMCA verification for WWMCA 
Boxes 22 and 29 using a Cloudsat cloud mask value of 30 for the cloud/no-cloud 
threshold. For comparison, WWMCA verification results from Cloudsat cloud mask 
value of 20 for cloud/no-cloud threshold are shown in Appendix C. The number of 
occurrences of hits, false alarms, misses, and correct rejections for the three cloud cover 
events—3: Definite-Cloud, 2: Probable-Cloud, and 1: No-Cloud—described in section 
2.F.3, were used to characterize WWMCA performance through formulas stated in 
section 2.F.4. The performance results are provided for each of the three cloud cover 
events, for the daytime data, for nighttime data, and for combined daytime and nighttime 
data (referred to as “All Day” results). These results are presented for each of the four 
months we analyzed (January, April, July, and October) and for the two WWMCA boxes 
we analyzed (22 and 29). Note that in the figures in this chapter, Definite-Cloucl events 
are sometimes referred to as Cloud events. Our emphasis in analyzing WWMCA was on 
its performance in analyzing Definite-Cloud and No-Cloud events, since these are of 
most interest to IC users of CDFS II products. However, we also present in the following 
sections the results for WWMCA performance in analyzing Probable-Cloud events, 
which, as shown in these sections, tends to be quite low. 

B. WWMCA BOX 22 

For WWMCA Box 22, Figures 12 and 13 show the number of occurrences of hits 

(A), false alarms (B), misses (C), and correct rejections (D) for each cloud cover event, 

and Figure 14 shows the marginal distribution of the cloud events, for the four months. 

Figures 12 and 13 show that the highest number of occurrences for each cloud event was 

in the correct rejections category. No-Cloud events had the largest number of hits in each 

month and Definite-Cloud events had the second highest. The marginal distributions of 

cloud events for both WWMCA and Cloudsat (Figure 14) show a general decline 

(increase) in Definite-Cloud (No-Cloud) events from January through October. The 
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information in Figures 12-14 is useful in assessing the number and percentage of 
opportunities that WWMCA had to correct analyze the three cloud events (Figure 14) and 
how well WWMCA did in analyzing those events (Figures 12-14). For example, Figure 
14 shows that in July and October there were high percentages of No-Cloud events, but 
relatively low percentages of Definite-Cloud events. This in turn indicates that in July 
and October: (a) No-Cloud events were relatively common and therefore perhaps 
relatively easy to correctly analyze; and (b) Definite-Cloud events were relatively rare 
and therefore perhaps relatively difficult to correctly analyze. 
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April Nighttime Occurrences WWMCA (22) - Cloudsat (30) 2010 


m 


■ Cloud 

■ ProbableCloud 

■ No Cloud 


60000 



■ Cloud 

■ ProbableCloud 

■ No Cloud 



April All Day Occurrences WWMCA (22) - Cloudsat (30) 2010 
120000 - 



■ Cloud 

■ ProbableCloud 

■ No Cloud 


Figure 12. Number of 2 x 2 contingency table occurrences in January and April 2010 for 
WWMCA Box 22 for all three cloud cover events: Definite-Cloud (blue), 
Probable-Cloud (red), and No-Cloud (green). On the horizontal axes: A 
indicates Hits, B indicates False Alarms, C indicates Misses, and D indicates 
Correct Rejections. The top panels shows the daytime occurrences, the 
middle panels shows the nighttime occurrences, and the bottom panels show 
the results for both daytime and nighttime occurrences. Results based on 
using a Cloudsat cloud mask value of 30 for the cloud/no-cloud threshold. 
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Figure 13. Number of 2 x 2 contingency table occurrences in July and October 2010 for 
WWMCA Box 22 for all three cloud cover events: Definite-Cloud (blue), 
Probable-Cloud (red), and No-Cloud (green). On the horizontal axes: A 
indicates Hits, B indicates False Alarms, C indicates Misses, and D indicates 
Correct Rejections. The top panels shows the daytime occurrences, the 
middle panels shows the nighttime occurrences, and the bottom panels show 
the results for both daytime and nighttime occurrences. Results based on 
using a Cloudsat cloud mask value of 30 for the cloud/no-cloud threshold. 
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Daytime Marginal Distributions of Cloud Cover Events for WWMCA 
(22) - Cloudsat (30) 2010 
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Nighttime Marginal Distributions of Cloud Cover Events for 
WWMCA (22) - Cloudsat (30) 2010 
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All Day Marginal Distributions of Cloud Cover Events for WWMCA 
(22) - Cloudsat (30) 2010 
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Figure 14. Marginal distributions of cloud cover events for WWMCA and Cloudsat for 
WWMCA Box 22: Definite-Cloud, Probable-Cloud, and No-Cloud). The 
calculation of the marginal distributions is shown in Table 7. The top panel 
shows the results for daytime data, the middle panel for nighttime data, and 
the bottom panel for both daytime and nighttime data. Results based on using 
a Cloudsat cloud mask value of 30 for the cloud/no-cloud threshold. 
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The POD for the cloud cover events are shown in Figure 15. Overall, WWMCA 
did the best for No-Cloud events year round—especially during the daytime and in July 
and October, when No-Cloud events were relatively common (Figure 14). In particular, 
WWMCA analyses of No-Cloud events were correct 85—95% of the time. But 
WWMCA analyses of Definite-Cloud events were correct about 57—62% of the time for 
winter, spring, and summer, and 45% of the time in the fall. The mean Daytime No- 
Cloud POD was 95% and the mean nighttime No-Cloud POD was 88%. The mean POD 
of Definite-Cloud events were considerably less: daytime ranged 36—64% and nighttime 
49—65%. The lowest Definite-Cloud POD occurred in fall daytime and summer 
nighttime. The POD for Definite-Cloud events are similar to but somewhat lower than 
those from UCAR (2008), which reported a POD for Cloud events of 64% (Chapter I, 
Section C). The relatively high (low) POD for No-Cloud (Definite-Cloud) events may be 
related to the abundance (scarcity) of No-Cloud (Definite-Cloud) events, especially in 
July and October (see prior discussion of Figure 14). 


39 



Daytime Hit Rate WWMCA (22) - Cloudsat (30) 2010 


Figure 15. 
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WWMCA probability of detection (POD), or hit rate (H), for WWMCA Box 
22 for all three cloud cover events: Definite-Cloud (blue), Probable-Cloud 
(red), and No-Cloud (green). The top panel shows the results for daytime 
data, the middle panel for nighttime data, and the bottom panel for both 
daytime and nighttime data. Results based on using a Cloudsat cloud mask 
value of 30 for the cloud/no-cloud threshold. 
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Figure 16 shows the WWMCA FAR for Box 22, which range from 5 to 40%. 
The FAR for Definite-Cloud was relatively low in winter and spring, but relatively high 
in summer and fall. The FAR for No-Cloud events showed the opposite evolution, from 
relatively high in winter and spring to relatively low in summer and fall. This is 
consistent with the occurrence and marginal distributions results (Figures 12-14), with 
the FAR for an event type being lower when the event type is common and higher when 
the event type is rare. The FAR results also correspond with the POD results, especially 
the low FAR for No-Cloud events in summer and fall that is associated with a high POD 
of 95%. The results shown in Figures 12-16 suggest that WWMCA did better in 
persistent, or common, cloud cover conditions—for example, better in analyzing 
Definite-Cloud events in winter and spring when those events were common, and better 
in analyzing No-Cloud events in summer and fall when those events were common. 


41 



Daytime FAR WWMCA (22) - Cloudsat (30) 2010 



Nightime FAR WWMCA (22) - Cloudsat (30) 2010 



• Cloud 

■ Probable Cloud 

* Ho Cloud 


All Day FAR WWMCA (22) - Cloudsat (30) 2010 



— ^^—Cloud 

■ Probable Cloud 
» Ho Cloud 


Figure 16. WWMCA false alarm ratios for WWMCA Box 22 for all three cloud cover 
events: Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud (green). 

The top panel shows the results for daytime data, the middle panel for 
nighttime data, and the bottom panel for both daytime and nighttime data. 
Results based on using a Cloudsat cloud mask value of 30 for the cloud/no¬ 
cloud threshold. 


The WWMCA PC results for Box 22 are shown in Figure 17. The three cloud 
cover events had similar performance, with better performance in the summer and fall 
than winter and spring. Day and nighttime PC were within approximately 15% of each 
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other throughout the year. Overall, the PC results for Definite-Cloud events were a little 
better than for No-Cloud events. The PC results are consistent with the POD and FAR 
results in indicating that WWMCA did well in persistent conditions. 


Daytime PC WWMCA (22) - Cloudsat (30) 2010 



-Cloud 

-Probabl** Cloud 
-Mo Cloud 


Jan Apr Jul Oct 

Nighttime PC WWMCA (22) - Cloudsat (30) 2010 



- Cloud 

- Probable Cloud 

- Mo Cloud 


Jan Apr Jul Oct 

All Day PC WWMCA (22) - Cloudsat (30) 2010 



-Cloud 

-Probable Cloud 
-Mo Cloud 


Apr 


Figure 17. WWMCA proportion correct (PC) for WWMCA Box 22 for all three cloud 
cover events: Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud 
(green). The top panel shows the results for daytime data, the middle panel 
for nighttime data, and the bottom panel for both daytime and nighttime data. 
Results based on using a Cloudsat cloud mask value of 30 for the cloud/no¬ 
cloud threshold. 
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The PC performance metrics gives equal credit for hits and correct rejections 
(Chapter II, Section F.4; Wilks 2006). Thus, we also calculated the threat score (TS) to 
focus the performance analysis on the hits. The TS results for Box 22 are shown in 
Figure 18 and reveal uniformly better No-Cloud performance than Definite-Cloud 
performance, with the best performance being for No-Cloud in the summer and fall 
(about 85%). Daytime TS for no-cloud was about 68% in winter and spring and 
increased to about 85% in summer and fall; TS for definite cloud was about 50% in 
winter and spring and decreased to 45% in summer and 34% in fall. Nighttime TS for 
no-cloud was about 65% in winter and spring and increased to 87% in summer and fall; 
definite cloud was 52% in winter and spring and increased to about 42% in summer and 
fall. The high TS for No-Cloud during summer and fall is consistent with the POD of 
95% (Figure 15) and the low FAR of 10% (Figure 16). The higher TS for Definite-Cloud 
events in winter and spring are related to Definite-Cloud events having been relatively 
common in those months (Figure 14). Taken together, these results support the idea that 
WWMCA performs: 

1. better when cloud conditions that are relatively persistent and/or common 

2. worse when analyzing cloud conditions that are relatively variable 
(spatially, temporally) and/or uncommon. 
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Figure 18. WWMCA threat scores (TS) for WWMCA Box 22 for all three cloud cover 
events: Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud (green). 

The top panel shows the results for daytime data, the middle panel for 
nighttime data, and the bottom panel for both daytime and nighttime data. 
Results based on using a Cloudsat cloud mask value of 30 for the cloud/no¬ 
cloud threshold. 
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Figure 19 shows the bias (B) and indicates that No-Cloud events were over 
analyzed in winter, spring, and fall, and were mostly unbiased in summer. Definite- 
Cloud events were consistently under analyzed, especially in the fall, with the exception 
of summer when the analysis of Definite-Cloud events was mostly unbiased. These 
results are consistent with the findings of the UCAR (2008) and AER (2010) studies. 
There is a larger bias for daytime Definite-Cloud and No-Cloud events in winter and fall, 
as opposed to spring and summer, and there is a generally lower bias for Definite-Cloud 
events at night. These results are also consistent with the results shown in Figures 15-18 
For example, the over (under) analysis of No-Cloud (Definite-Cloud) in October is 
consistent with the high (low) POD in that month. 
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Figure 19. WWMCA bias (B) for WWMCA Box 22 for all three cloud cover events: 

Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud (green). The top 
panel shows the results for daytime data, the middle panel for nighttime data, 
and the bottom panel for both daytime and nighttime data. The heavy dark 
line at B = 1 indicates an unbiased analysis. Values greater (less) than 1 
indicate a cloud cover event that was over (under) analyzed. Results based on 
using a Cloudsat cloud mask value of 30 for the cloud/no-cloud threshold. 
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The Heidke skill scores are shown in Figure 20 and show relatively little 
difference between the months or between day and night. The highest skill scores were 
associated with No-Cloud, ranging from about 50% in January night to 65% in October 
night. The HSS results for Definite-Cloud events are similar to but somewhat less than 
those for No-Cloud events. As with the other performance metrics, the poorest 
performance for Definite-Cloud events is in October. The HSS for Probable-Cloud 
events are considerably lower than for the other two events, ranging from 8% to 13% 
throughout the year. The HSS results are consistent with the POD, FAR, PC, and TS 
results. However, it is interesting to note that HSS shows relatively small differences in 
July and October between No-Cloud and Definite-Cloud, unlike the case, in general, for 
POD, FAR, and TS. This may be because HSS gives more (less) credit for good 
performance in analyzing uncommon (common) events—and in July and October; 
Definite-Cloud (No-Cloud) events were uncommon (common) (see Figure 14). 
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Figure 20. WWMCA Heidke skill score (HSS) for WWMCA Box 22 for all three cloud 
cover events: Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud 
(green). The top panel shows the results for daytime data, the middle panel 
for nighttime data, and the bottom panel for both daytime and nighttime data. 

The heavy dark line at the zero value indicates that the WWMCA is 
equivalent to reference analyses (random analyses statistically independent of 
observations). A perfect WWMCA cloud analysis would have a value of one. 
Negative values indicate WWMCA performs worse than reference analyses. 
Results based on using a Cloudsat cloud mask value of 30 for the cloud/no¬ 
cloud threshold. 
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c. 


WWMCA BOX 29 


For Box 29, we analyzed points north of 70° N as being in polar night during all 
of January and all of April, and in polar day during all of July and all of October. This is 
an incorrect way to analyze the April and October data that arose due to an error in the 
analysis coding. Thus, for the April and October results, readers should focus their 
attention on just the All Day results. 

Figures 21 and 22 show the number of occurrences of hits (A), false alarms (B), 
misses (C), and correct rejections (D) for each cloud cover event, and Figure 23 shows 
the marginal distribution of the cloud events, for the four months. Note that there was a 
more even distribution of the three cloud event types for Box 29 (Figures 21 and 22) than 
for Box 22 (Figures 12 and 13). In particular, the marginal distribution of cloud events 
for WWMCA and Cloudsat (Figure 23) show that neither Definite-cloud nor No-Cloud 
events dominated in 2010 in Box 29 as opposed to Box 22. Thus, Box 29 in 2010 
represented more mixed and variable cloud conditions than did Box 22 in 2010. Box 29 
for 2010 may also have presented WWMCA with a more challenging analysis problem 
than did Box 22, since Box 29 showed less persistence of any one cloud event type. Note 
too that WWMCA understated the percentage of Definite-Cloud events, and overstated 
the number of No-Cloud events, except during July (Figure 23). WWMCA also 
overstated the percentage of Probable-Cloud events in all months. 
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Figure 21. Number of 2 x 2 contingency table occurrences in January and April 2010 for 
WWMCA Box 29 for all three cloud cover events: Definite-Cloud (blue), 
Probable-Cloud (red), and No-Cloud (green). On the horizontal axes: A 
indicates Hits, B indicates False Alarms, C indicates Misses, and D indicates 
Correct Rejections. The top panels shows the daytime occurrences, the 
middle panels shows the nighttime occurrences, and the bottom panels show 
the results for both daytime and nighttime occurrences. Results based on 
using a Cloudsat cloud mask value of 30 for the cloud/no-cloud threshold. 
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Figure 22. Number of 2 x 2 contingency table occurrences in July and October 2010 for 
WWMCA Box 29 for all three cloud cover events: Definite-Cloud (blue), 
Probable-Cloud (red), and No-Cloud (green). On the horizontal axes: A 
indicates Hits, B indicates False Alarms, C indicates Misses, and D indicates 
Correct Rejections. The top panels shows the daytime occurrences, the 
middle panels shows the nighttime occurrences, and the bottom panels show 
the results for both daytime and nighttime occurrences. Results based on 
using a Cloudsat cloud mask value of 30 for the cloud/no-cloud threshold. 
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Daytime Marginal Distributions of Cloud Cover Events for WWMCA 
(29) - Cloudsat (30) 2010 
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Nighttime Marginal Distributions of Cloud Cover Events for 
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All Day Marginal Distributions of Cloud Cover Events for WWMCA 
(29) - Cloudsat (30) 2010 
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Figure 23. Marginal distributions of cloud cover events for WWMCA and Cloudsat for 
WWMCA Box 29: Definite-Cloud, Probable-Cloud, and No-Cloud). The 
calculation of the marginal distributions is shown in Table 7. The top panel 
shows the results for daytime data, the middle panel for nighttime data, and 
the bottom panel for both daytime and nighttime data. Results based on using 
a Cloudsat cloud mask value of 30 for the cloud/no-cloud threshold. 
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The POD for the cloud cover events are shown in Figure 24. Note that the All 
Day results show PODs for: (1) No-Cloud events were 71-78% in winter and spring, but 
51-60% in summer and fall; (2) Definite-Cloud events were 42% in winter and spring, 
64% in summer, and 41% in fall; and (3) Probable-Cloud events were 15%-29%, with an 
increasing trend throughout the year. These POD results were considerably lower than 
those for Box 22 (Figure 15). The Definite-Cloud results are also lower than the 64% 
POD for cloud reported in UCAR (2008). 
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Figure 24. WWMCA probability of detection (POD), or hit rate (FI), for WWMCA Box 
29 for all three cloud cover events: Definite-Cloud (blue), Probable-Cloud 
(red), and No-Cloud (green). The top panel shows the results for daytime 
data, the middle panel for nighttime data, and the bottom panel for both 
daytime and nighttime data. Results based on using a Cloudsat cloud mask 
value of 30 for the cloud/no-cloud threshold. 
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Figure 25 show that Definite-Cloud events had an All Day FAR of about 22% to 
34% throughout the year, except for summer when the FAR was 41%. This summer 
increase in FAR is consistent with Figure 23, which shows the marginal distribution of 
WWMCA exceeded the marginal distribution of Cloudsat in July. The All Day FAR for 
No-Cloud events ranged from 33% to 46% throughout the year, except in October when 
it increased to 69%. Note that Figure 23 shows that the WWMCA No-Cloud marginal 
distribution was double that of Cloudsat in October. Note that the FAR values for Box 
29 were considerably higher than for Box 22 (Figure 16). 



Jan Apr Jul Oct 



Jan Apr Jul Oct 



Jan Apr Jul Oct 


Figure 25. WWMCA false alarm ratios for WWMCA Box 29 for all three cloud cover 
events: Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud (green). 

The top panel shows the results for daytime data, the middle panel for 
nighttime data, and the bottom panel for both daytime and nighttime data. 
Results based on using a Cloudsat cloud mask value of 30 for the cloud/no¬ 
cloud threshold. 
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The WWMCA PC results for Box 29 are shown in Figure 26. The All Day 
Definite-Cloud and No-Cloud events had similar scores, about 64%, throughout the year, 
although Definite-Cloud decreased to 53% in fall. The lower PC for Definite-Cloud in 
October is consistent with the lower Definite-Cloud POD values in October (Figure 24). 
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Figure 26. WWMCA proportion correct (PC) for WWMCA Box 29 for all three cloud 
cover events: Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud 
(green). The top panel shows the results for daytime data, the middle panel 
for nighttime data, and the bottom panel for both daytime and nighttime data. 
Results based on using a Cioudsat cloud mask value of 30 for the cloud/no¬ 
cloud threshold. 
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The TS results for Box 29 are shown in Figure 27 and reveal similar No-Cloud 
and Definite-Cloud performance. All Day No-Cloud TS was slightly better in winter 
spring and All Day Definite-Cloud TS was slightly better in summer and fall. Box 29 
had considerably lower TS scores than Box 22 (Figure 18), which is consistent with the 
lower POD (Figure 24) and higher FAR (Figure 25) for Box 29. These results suggest 
that WWMCA performance is worse when there is relatively high variability and low 
persistence in the cloud event types (Figures 21-27 for Box 29) and better when there is 
relatively low variability or high persistence (Figures 12-20 for Box 22). 
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Figure 27. WWMCA threat scores (TS) for WWMCA Box 29 for all three cloud cover 
events: Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud (green). 

The top panel shows the results for daytime data, the middle panel for 
nighttime data, and the bottom panel for both daytime and nighttime data. 
Results based on using a Cloudsat cloud mask value of 30 for the cloud/no¬ 
cloud threshold. 


Figure 28 shows the bias (B) results for Box 29. Definite-Cloud (No-Cloud) 
events were substantially under-analyzed (over analyzed) in winter, spring and fall, 
consistent with the marginal distribution results (Figure 23). The four month average All 
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Day Definite-Cloud (No-Cloud) bias was approximately 1.5 (-0.6). The overall patterns 
in Figure 28 are similar to those for Box 22 (Figure 19) but with larger bias magnitudes 
than for Box 22. 


Figure 28. 
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WWMCA bias (B) for WWMCA Box 29 for all three cloud cover events: 
Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud (green). The top 
panel shows the results for daytime data, the middle panel for nighttime data, 
and the bottom panel for both daytime and nighttime data. The heavy dark 
line at B = 1 indicates an unbiased analysis. Values greater (less) than 1 
indicate a cloud cover event that was over (under) analyzed. Results based on 
using a Cloudsat cloud mask value of 30 for the cloud/no-cloud threshold. 
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The Heidke skill scores for Box 29 are shown in Figure 29 and are similar for day 
and night, and for all months. The HSS values are in the 11% to 32% range for Definite- 
Cloud and No-Cloud, and -4% to 9% range for Probable-Cloud. These relatively low 
HSS values, compared to Box 22, are consistent with the lower POD, PC, TS and the 
higher FAR for Box 29 than for Box 22. Note that the Box 29 HSS values show 
relatively small differences through the year between No-Cloud and Definite-Cloud 
events, unlike the case, in general, for POD, FAR, and TS. Similar to Box 22, this may 
be because HSS gives more (less) credit for good performance in analyzing uncommon 
(common) events (Figure 23). 

Positive HSS indicates skill above a random analysis (see Chapter II, Section 
F.4). However, HSS values that don’t exceed 32% for Box 29 (Figure 29) or 65% for 
Box 22 (Figure 20) are problematic, given that WWMCA is a near real-time analysis (as 
opposed to a non-zero lead forecast) that is based on many real-time observational data 
sources. Unfortunately, there do not appear to be any other studies of WWMCA that 
have reported HSS results or other skill scores that compare WWMCA to a reference, or 
benchmark, analysis. 
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Figure 29. WWMCA Heidke skill score (HSS) for WWMCA Box 29 for all three cloud 
cover events: Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud 
(green). The top panel shows the results for daytime data, the middle panel 
for nighttime data, and the bottom panel for both daytime and nighttime data. 

The heavy dark line at the zero value indicates that the WWMCA is 
equivalent to reference analyses (random analyses statistically independent of 
observations). A perfect WWMCA cloud analysis would have a value of one. 
Negative values indicate WWMCA performs worse than reference analyses. 
Results based on using a Cloudsat cloud mask value of 30 for the cloud/no¬ 
cloud threshold. 
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D. SUMMARY OF RESULTS 


The results show that WWMCA performance was: (1) better for the southwest 
Asia region where there was more persistence in cloud cover events (especially No-Cloud 
events); and (2) worse for the western Russia - Barents Sea region where there was more 
variability in cloud cover events. This makes some sense, because persistent conditions 
tend to be easier to analyze. The marginal distribution and HSS results provide a good 
summary of how differences in performance were associated with differences in cloud 
cover event persistence. The relatively good performance for the southwest Asia region 
may be due to the relatively clear skies that are typical of that region, and that 
characterized the region during much of 2010, especially in summer and fall. Another 
factor that may have led to the relatively low performance for the western Russia - 
Barents Sea region may be the higher pixel ages for the higher latitudes in which polar 
orbiting satellites are a main source of cloud information for WWMCA (see Chapters II 
and IV for more on this topic). 

There were also notable differences in the results for the different months, and for 
day and night (e.g., in the All Day POD and HSS results for Box 29, and in the Box 22 
bias results for day and night). 

Overall, the WWMCA performance results appear to be at least somewhat 
problematic, since: (1) WWMCA products are near real-time analyses based on many 
near real-time observational data sources; and (2) WWMCA is used to initialize cloud 
forecast models that are used in IC planning. 
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IV. SUMMARY, CONCLUSIONS AND RECOMMENDATIONS 


A. SUMMARY 

We conducted a verification of the WWMCA cloud analysis product generated by 
CDFS II that is used in the planning of overhead imagery collection. Our independent 
verification data source was data from the Cloudsat atmospheric sounder, a satellite- 
borne radar system designed to sense condensed cloud particles. We looked at the cloud 
analyses of two WWMCA boxes that are of high interest to the national intelligence 
community and that differ in their meteorological characteristics. One box was centered 
in western Russia and the nearby polar region, and the other box was in the mid-latitudes 
and contained mountain, desert, ocean, and coastal regions. We analyzed one year of 
data for these two WWMCA boxes to test a system for routinely monitoring WWMCA 
performance. 

Our goal was to analyze the performance of WWMCA for a range of cloud cover 
events, times, and locations, and using a range of performance metrics. To do so, we had 
to develop a method to match the data sets and to ensure an adequate amount of ground 
truth measurements fell within each WWMCA cell to confidently verify that cell’s cloud 
analysis. The WWMCA data set we used was provided by the 16 th Weather Squadron. 
The Cloudsat data was downloaded from the Cloudsat Data Processing Center website. 
After overcoming the differences in temporal and spatial resolutions of the data, we were 
able to calculate metrics that provide useful information about the performance of 
WWMCA. 

B. CONCLUSIONS 

Our performance metrics showed that WWMCA performed better under 
persistent cloud cover situations than more variable cloud situations. For instance, 
WWMCA had a high POD for No-Cloud events when there were persistently few clouds. 
For mid- and high latitudes, Definite-Cloud events were under analyzed and No-Cloud 
events were over analyzed, except for the summer when both events were unbiased. The 
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Heidke skill scores for mid-latitude cloud analyses were considerably higher than those 
for high latitude cloud analyses—for example, high HSS values in the mid-latitudes of 
about 60% versus about 30% for the high latitudes. Performance in analyzing Probable- 
Cloud events was especially poor in both regions. For example: (1) HSS for Probable- 
Cloud events ranged between -5% and 15% in both mid- and high-latitudes; and (2) the 
Heidke Skill Scores were less than 14% for Boxes 22 and 29 year round. 

The better performance of WWMCA in the mid-latitudes could be accounted for 
by the higher temporal resolution of the geostationary satellites that provide the majority 
of the observations at those latitudes (evident in the pixel ages listed in Table 8). The 
mean pixel ages for Box 22 are 34-36 minutes and have a maximum standard deviation 
of 11 minutes. Conversely, the low temporal resolution of observational imagery from 
polar-orbiting satellites could explain why the high latitude cloud analyses had lower 
skill. The mean pixel ages for Box 29 are on the order of three hours and standard 
deviations are about two hours (Table 8). 


Table 8. Pixel age information for satellite observations used in the WWMCA 
development process. Means and standard deviations calculated from the 
pixel age data in the WWMCA data set. Negative minutes are due to a 
portion of the satellite imagery coming in after the :00/:30 time hack (Connor 

2012, personal communication). 




Box 22 



Box 29 


2010 

Min 

Age 

Max 

Age 

Mean 

Age 

Std. Dev. 

Min 

Age 

Max 

Age 

Mean 

Age 

Std. Dev. 

Jan 

16 

154 

34 

7.8 

24 

503 

156 

109 

Apr 

12 

138 

35 

9.5 

22 

626 

166 

117 

Jut 

14 

138 

36 

11 

21 

613 

177 

115 

Oct 

-7 

153 

36 

9.3 

-3 

497 

171 

121 


We determined that WWMCA performance is relatively insensitive to reasonable 
choices of the Cloudsat cloud masking threshold (values between 20 and 30), as shown 
by a comparison of our results in Chapter III to those in Appendix C. 

The WWMCA performance results that we developed are generally lower than to 
those reported in the available documentation for prior studies (e.g., UCAR (2008). 
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However, reports of performance results from prior studies are sparse (see Chapter I, 
Section C). In addition, we verified WWMCA performance for larger areas and a wider 
range of months than most prior studies, and we verified WWMCA using a wider range 
of metrics. The paper that summarized most of the findings of the prior studies, NGIS 
(2011), disclosed only the POD and not the FAR or other scores that might have been 
generated in those studies. Of course, determining performance based only on POD or 
any other single metrics, is very problematic (cf. Wilks 2006). 

C. RECOMMENDATIONS 

Future analyses of WWMCA results and its operational impacts on imagery 
collection are necessary to truly assess its performance. We recommend that future 
studies expand on the regions, months, and years examined to evaluate the regional-scale, 
global-scale, and long-term performance of WWMCA. The analysis of many years of 
data would help address issues raised by unusual events, such as the record high 
temperatures and extensive wildfires in western Russia in summer 2010 (which may have 
had an impact on both Cloudsat observations and WWMCA analyses in 2010). 

Our results raised the question: Is WWMCA more accurate when pixels ages are 
small? We recommend an analysis that attempts to answer that question by testing the 
sensitivity of WWMCA performance to pixel age, especially the performance at high 
latitudes. 

A future study should test the sensitivity of our assumptions, as discussed at the 
end of Chapter II. For example, we recommend a future study to test the minimum 
Cloudsat pixels necessary to confidently represent a WWMCA cell. Recall in Chapter II 
that we assumed that a minimum of six Cloudsat pixels (25% coverage of a WWMCA 
cell) could adequately represent a WWMCA cell. 

Future studies should use a more dynamic method for determining sunrise/ and 
sunset times method to flag the data as either day or night. Recall in Chapter II that we 
chose the mean monthly sunrise and sunset times or a point close to the center of a 
WWMCA box (Table 6) to represent that entire WWMCA box, and that for the polar 
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region (> 70° N), we flagged data as either polar night or polar day based on the season. 
These approximations were used to reduce the data processing time but also introduced 
some errors in assigning the time of day to the data that should be removed in future 
studies that focus on distinguishing day and night performance. Applying a 
sunrise/sunset library to the data matching stage, such as the one used in Selin (2011, 
personal communication) study, would help remove these errors. 

We found that WWMCA performs better in persistent cloud cover situations—as 
indicated, for example, by our HSS results. We recommended that a future study 
compare the skill of WWMCA to that of persistent analysis to test the validity of this 
finding and to better assess WWMCA skill. 

Additionally, we recommend an in-depth study that will exploit the methods 
developed in our study to analyze large amounts of WWMCA data and show the long¬ 
term performance of WWMCA. One benefit of such a study is that it would help 
determine the value added by past and future changes to CDFS n. This study should 
look at long time periods, various WWMCA boxes from both hemispheres, and a range 
of climate conditions (e.g., high and low cloud tops; high and low cloud variability; high 
and low cloud amounts; land, ocean, snow, ice, mountain, and desert surfaces; tropical, 
mid-latitude, and polar regions; etc.) to measure the overall and specific aspects of 
WWMCA performance. 

The operational impacts of cloud analyses and forecasts are being tested in an 
ongoing study at NPS by Lieutenant Brian Moore (classified thesis expected to be 
published in September 2012, Figure 1). We recommend a review of his study when 
completed to gain insight into the operational significance of WWMCA in imagery 
collection. 

For access to the data sets and computer coding used in this study, contact Dr. 
Tom Murphree in the Department of Meteorology at the Naval Postgraduate School 
(murphree@nps.edu) . 


"Who is wise enough to number all the clouds...?" 
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APPENDIX A. CLOUD DEPICTION FORECAST SYSTEM (CDFS) 

II PROCESSING LEVELS 


Satellite and conventional observations (surface observations and upper air 
soundings) undergo a four level process to be merged into a global cloud analysis (Figure 
30). Level one is data calibration, level two classifies each pixel into cloudy or clear, 
level three applies cloud layering and typing, and level four consists of merging the 
separate analyses into one global analysis (HQ AFWA/DNXM 2011). 



Figure 30. Illustration of the four processing levels within Cloud Depiction Forecast 
System II. Observations are received from meteorological satellites, 
conventional observations, and global analysis from various models. These 
observations are merged into one global cloud analysis that is used to initiate 
cloud forecast models. Shapes are defined as: rectangles are processes; 
rounded rectangles are inputs; ovals are products; and snipe same side corner 

rectangles are cloud models. 
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A. 


LEVEL 1 


Level one processing consists of data ingestion and calibration. Satellite 
telemetry transmissions are received by AFWA’s Satellite Data Handling System located 
at Offutt Air Force Base in Nebraska. Downlinked satellite data is encoded and must be 
decoded. The decoded data reveal physical parameters for radiance measurements 
received by the detectors on the sensor’s focal plane array and are placed in the Sensor 
Data Records (SDR). Satellite imagers’ focal plane array consist of detectors that 
represent the pixels, which measure the radiance received from reflected or emitted 
energy within Earth’s atmosphere. CDFS II use the data from the satellite’s visible and 
infrared channels. A calibration step for the infrared data converts the measured emitted 
energy into either radiance or brightness temperature. Brightness temperature is the 
temperature of an object if it was radiating as a black body. The brightness temperature 
is the parameter required by CDFS II to make the analysis. Reflectance values are 
measured from the satellite’s visible channels are used directly in the algorithms (HQ 
AFWA/DNXM 2011). 

B. LEVEL 2 

Level two is where cloud detection occurs. Each sensor has its own tailored 
algorithms designed to optimize their instrument’s ability to exploit measurements made 
in different channels in an attempt to distinguish cloud from clear scene. DMSP’s 
Operational Line Scanner (OLS) sensor, has the highest spatial resolution, but only two 
broadband channels (one visible and one infrared), whereas, NOAA POES’ Advanced 
Very-High Resolution Radiometer (AVHRR), have six narrowband channels (one 
visible, one near-infrared, and four infrared). DMSP’s OLS cloud detection is 
accomplished by comparing a pixel’s brightness temperature to a cloud-free referenced 
pixel’s brightness temperature. If the pixel is determined to be cloud free, its brightness 
temperature is then used as the clear-scene brightness temperature for all other pixels in 
the frame. To determine a cloud-filled pixel, the observed brightness temperature of the 
said pixel is compared to the predicted clear-scene brightness temperature. The 
difference in magnitude of brightness temperature determines if the pixel is cloud-filled 
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or cloudfree. A similar method is used for the visible channel when available. Threshold 
values are used to determine the cutoff between cloudy and clear pixel (HQ 
AFWA/DNMX 2011). 

The algorithm for NOAA’s AVHRR exploits the multispectral properties of the 
sensor to identify cloud or clear scene. The strength of the AVHRR is its six channels 
designs, which are listed in Table 9 with their respective wavelengths and typical use. 
Each channel differs in their sensitivity to reflectivity and emissivity properties of clouds 
and clear terrestrial surfaces. In addition to the sensors data, the algorithm uses dear- 
scene characterizations of the terrestrial background. The algorithm utilizes various 
techniques to include straight threshold type algorithms, inter-channel comparisons and 
spectral comparisons between the terrestrial surface and satellite data. A suite of twelve 
tests are used to characterize the different spectral characteristics of clouds and 
background surfaces to determine cloud-filled or clear scene. Each test is based on one 
or more specific spectral signatures that compare the radiance measurement of one or 
more channels, and fall into either cloud tests or background tests categories (HQ 
AFWA/DNXM 2011). 
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Table 9. 


The channels below are from the AVHRR/3 sensor. The AVHRR is a 
radiation-detection imager that can be used for remotely determining cloud 
cover and the surface temperature. Note that the term surface can mean the 
surface of the Earth, the upper surfaces of clouds, or the surface of a body of 
water. This scanning radiometer uses 6 detectors that collect different bands 
of radiation wavelengths as shown below. Measuring the same view, this 
array of diverse wavelengths, after processing, permits multi spectral analysis 
for more precisely defining hydrologic, oceanographic, and meteorological 
parameters. Comparison of data from two channels is often used to observe 
features or measure various environmental parameters. The three channels 
operating entirely within the infrared band are used to detect the heat 
radiation from and hence, the temperature of land, water, sea surfaces, and 
the clouds above them. Table based from 
NO A AS IS (2011). 


Channel 

number 

Resolution at nadir 
(km) 

Wavelength (pm) 

Typical use 

1 

1.09 

0.58-0.68 

Daytime cloud and surface mapping 

2 

1.09 

0.725-1.00 

Land-water boundaries 

3a 

1.09 

1.58-1.64 

Snow and ice detection 

3b 

1.09 

3.55-3.93 

Night cloud mapping, sea surface temperature 

4 

1.09 

10.30-11.30 

Night cloud mapping, sea surface temperature 

5 

1.09 

11.50-12.50 

Sea surface temperature 


There are nine cloud tests and three background tests, which are summarized in 
Table 10. Different tests are used to identify clouds under different conditions. The low 
clouds and fog test for solar-illuminated data is used to identify water droplets based low- 
level cloud when the scene is illuminated by sunlight, and the non-illuminated test is used 
during nighttime. Since no one test will identify all the clouds in a scene, the cloud tests 
must be used in combination to accurately identify all cloud-filled pixels. The 
background tests are unique to the AVHRR algorithm, which exploits the multispectral 
characteristics of AVHRR data, to identify snow and ice, desert and sun glint 
backgrounds. These tests are essential because clouds and surface features often exhibit 
similar spectral signatures in the visible spectrum, however; a positive result from these 
tests does not automatic mean a cloudfree pixel. These tests identify suspected visible 
data; the infrared cloud tests must still be applied to determine a cloud-filled pixel (HQ 
AFWA/DNXM 2011). 
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Table 10. 


Cloud analysis test for the NOAA AVHRR level 2 algorithm. Table from 

HQ AFWA/DNXM (2011). 





Visible 

IR Data 

Test Type 

Test Name 

Major Identifying Class 

Data 



Low cloud and fog test for solar-illumination 
data 

Solar-illumination liquid water cloud 


X 


Precipitation cloud test 

Cumulonimbus 

X 

X 

Cloud 

Thin cirrus cloud test for solar-illumination 
data 

Solar-illumination ice cloud 

X 

X 


Visible brightness ratio 

Solar-illuminated liquid water cloud 

X 



Single channel visible brightness test 

Solar-illuminated liquid water cloud 

X 



Cold cloud test 

Mid- to high-level optically thick water 
and ice cloud 


X 


Cirrus cloud test 

High-level ice cloud 


X 


Fog, low stratus test for non-solar-illuminated 
data 

Non-solar-illuminated liquid water cloud 


X 


Thin cirrus cloud for non-solar-illuminated 
data 

Non-solar-illuminated ice cloud 


X 


Sun glint background test 

Water surfaces exhibiting specular 

X 

X 

Background 


reflection 




Desert background test 

Highly reflective non-vegetated land 
surfaces 

X 

X 


Snow/ice cover background test 

Highly reflective snow or ice covered 
land and water surfaces 

X 

X 


There are three algorithms used to detection clouds from geostationary satellite 
data. Geostationary satellites have a high temporal resolution but the spatial resolution is 
degraded due to its altitude (-36,000 km). The first algorithm takes advantage of the 
high temporal resolution to identify cloud-filled pixels by testing for rapid changes in 
brightness temperature and reflectance values in pixels representing the same 
geolocation. The pixels that exhibit changes in radiance values greater than the amount 
expected for clear scene from frame to frame are identified as cloud-filled. The second 
algorithm is a dynamic threshold algorithm that identifies cloud with similar 
characteristics. Cloud-filled pixels identified through the temporal difference algorithm 
are processed by the dynamic threshold algorithm. The dynamic threshold algorithm 
identifies maximum and minimum brightness temperatures or reflectance within a grid 
cell, which are used to define threshold values for cloud-filled and cloud-free pixels 
remaining within the grid cell. The third algorithm uses a series of spectral 
discrimination tests similar to the OLS and AVHRR spectral tests. Not all geostationary 
satellite data are the same, so a different set of tests may need to be ran for each satellite 
system. For instance, METOSAT platforms have different spectral channels than GOES 
(HQ AFWA/DNXM 2011). Tables 11 and 12 summarize the spectral channels for 
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METEOSAT and GOES, respectively. The resultant dataset, generated for each satellite 
data source, is called Cloud Data Records (CDR). 


Table 11. Spectral channels and bandwidth for METEOSAT satellites. Table from 

EUMETSAT (2011). 


Spatial Resolution 

Channel _ Wavelength (pm) _ at nadir (km) _ Remarks 

VIS 0.6 0.56-0.71 3 

VIS 0.8 0.74-0.88 3 

IR 1.6 1.50-1.78 3 

IR 3.9 3.48-4.36 3 


IR 8.7 

8.30-9.10 

3 

IR 10.8 

9.80-11.80 

3 

IR 12.0 

11.00-13.00 

3 

WV 6.2 

5.35-7.15 

3 

WV 7.3 

6.85-7.85 

3 

IR 9.7 

9.38-9.94 

3 

IR 13.4 

12.40-14.40 

3 

HRV 

0.5-0.9 

1 


Similar to AVHRR 
Similar to AVHRR 
Similar to AVHRR 
Similar to AVHRR 
New 

Similar to AVHRR 

Similar to AVHRR 

Water vapor channel 

Water vapor channel 

Ozone absorption channel as on HIRS 

C02 absorption channel as on GOES-VAS sounder 

High Resolution Visible (HRV); Broadband visible channel 


Table 12. GOES imager channels. Table from GOES Imager Channel Notation (2011). 


Channel 

number 

Resolution at nadir 
or IFOV* (km) 

Wavelength 

(pm) 

Remarks 

1 

1 

0.55-0.75 

Visible 

2 

4 

3.80-4.00 

Shortwave Infrared 

3 

8 

6.50-7.00 

Moisture 

4 

4 

10.20-11.20 

Infrared 1 

5 

4 

11.50-12.50 

Infrared 2 


* Instantaneous Field of View 


C. LEVEL 3 

Level three is where the satellite pixels are gridded onto AFWA’s standard Polar- 
stereographic grid at “16 th mesh” with a horizontal resolution of 24 km (true at 60° 
latitude). Pixels are assembled into the 16 th mesh grid cells by computing the coordinates 
that correspond to the latitude and longitude of each pixel (Hoke et al., 1981 Rev. March 
1985). A detailed description of the Polar-stereographic grid is provided in Map 
Projections and Grid System for Meteorological Applications, AFGWC Technical Notes 
79/003 (Hoke et al., 1981 Rev. March 1985). The cloud layers are identified through the 
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Long Wave Infrared (LWIR) brightness temperature data contained within each grid cell. 
A clustering algorithm clusters pixels of similar brightness characteristics to identify 
potential layer separations. Statistical procedures are applied to the grid cell to limit the 
identified layers to four. Once the layers are identified, cloud top temperatures are 
compared against vertical temperature information from the Nation Centers for 
Environmental Prediction (NCEP) Global Forecast System (GFS) model to assign a 
cloud top height. The cloud top height and temperature information is used with the 
visible/LWIR-count variance, from the background surface temperature model employed 
in level two, to assign each layer to one of nine different cloud types listed in Table 13. 
Along with each derived cloud type is a climatological cloud thickness that is subtracted 
from the cloud top height to determine the cloud base height (HQ AFWA/DNXM 2011). 
At the end of level three, each satellite family (e.g., DMSP, NOAA, Geostationary) have 
a common gridded cloud mask that consists of cloud fraction up to four layers. The 
cloud masking also includes cloud type, and cloud top/base heights. These datasets are 
called Gridded Data Records (GDR). 


Table 13. WWMCA default cloud thickness according to height. Cloud thickness is 
based on climatology. Table from HQ AFWA/DNXM (2011). 


Type Code 

Cloud Type 

Thickness (m) 

1 

Cumulonimbus (Cb) 

6300 

2 

Strata (St) 

600 

3 

Stratocumulus (Sc) 

1800 

4 

Cumulus (Cu) 

1200 

5 

Altostratus (As) 

1200 

6 

Nimbostratus (Ns) 

2100 

7 

Altocumulus (Ac) 

1800 

8 

Cirrostratus (Cs) 

1800 

9 

Cirrus (Ci) 

900 


Fevel three processing also includes hourly global surface and upper air based 
data, METARS or SYNTOPIC type formats, which contain fractional cloud coverage and 
cloud base heights from the World Meteorological Organization. These conventional 
observations are combined with the satellite data to determine the cloud mask, cloud 
type, and cloud top/base heights. 
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D. LEVEL 4 


Level four is where the satellite family GDRs and conventional surface 
observations are merged into a single global analysis of cloud cover information. One 
problem that arises in level four is that the independent gridded analyses have different 
valid times because the satellites input their data into CDFS II at different times. Each 
independent gridded analysis has strengths and weaknesses. For example, the polar- 
orbiters (DMSP and NOAA satellites) derived analyses have greater accuracy from the 
spatial resolution (polar satellites are in a lower orbit, -800 km); however, the temporal 
resolution is course, usually passing over a particular region one or two times a day. 
Geostationary satellites analyses have a finer temporal resolution, every 30 minutes, but 
spatial resolution, or instantaneous field of view (IFOV), varies from 1 to 8 km 
depending on the channel. See the resolution at nadir column in Table 12 for each 
channel’s IFOV. The timeliness and accuracy of the observations is a major concern 
when merging the data into a one global analysis (HQ AFWA/DNXM 2011). 

Integration of total cloud amounts supersedes integration of layered quantities 
since total cloud fraction estimates are more accurate than individual layer fraction due to 
the sample size of total cloud amount is far greater than the layered cloud amounts (HQ 
AFWA/DNXM 2011). Bartlett (2009) explains how the total cloud amount analysis 
works: 


Total cloud fraction is then set to either 100 or 0 percent, respectively. If 
neither analysis is completely cloud-filled or completely cloud-free, then 
the error for each analysis is estimated. The estimated errors for the 
analyses are compared to one another to see if the most recent analysis 
also has the lowest estimated error. Optimum interpolation (OI) occurs 
when one analysis cannot be chosen as the most accurate. OI maintains a 
blended estimate of total cloud fraction from multiple input analyses. 
Weighting functions for the OI are based on the estimated analysis errors 
which are computed for each individual analysis. Analysis errors are 
defined as an initial analysis error plus an additional error growth function 
which grows linearly with time. The error growth function is a tunable 
parameter that analysts can adjust to correct for inconsistencies. 

When total cloud amount is completed the other cloud parameters are merged. 

Rules applied, to determine which analysis is superior, in the layered analysis are similar 
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to the total cloud amount; however, if multiple timely analyses have 100% cloud cover or 
it is determined that an 01 technique is necessary. The integration of layered cloud 
amounts undergoes a more extensive algorithm. Most likely the individual analyses will 
have varying vertical distributions of cloud and cloud type due to the differences in 
sensor characteristics for each satellite family. The more complex algorithm determines 
which analyses is the most accurate and designates that analyses as the master template 
for which all other timely analyses are merged on. This process impacts discrete values 
such as the number of cloud layers and cloud types because these when integrated they 
will assume the values of the master template. The 01 procedure is used for varying 
layered cloud faction and cloud top temperature. The 01 process combines layers that 
closely match in cloud top temperatures and determines the layered cloud fraction. 
Special cloud algorithms have been designed for certain satellite sensors to enhance 
detection of low level stratus and cirrus. These special-case clouds are verified against 
the integrated analysis to be certain that the analysis is accurate, and are effective in 
showing the persistence of the observations in the subsequent integration analysis (HQ 
AFWA/DNXM 2011). 

All the output variables are placed in a GriB file (a collection of individual self- 
containing records, and the individual records themselves can stand alone as meaningful 
data) and is published as the Worldwide Merge Cloud Analysis (WWMCA). 
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APPENDIX B. SPREADSHEET STRUCTURE AND FORMULAS USED TO VERIFY WWMCA WITH 

CLOUDSAT 


Sheet 

Section 

Column 

Name 

Formula 

Remarks 



A 

WWMCA valid time 


Imported from CSV file 



B 

Cloudsat time 


Imported from CSV file 



C 

Cloudsat Lat 


Imported from CSV file 



D 

Cloudsat Lon 


Imported from CSV file 



E 

WWMCA Lat 


Imported from CSV file 



F 

WWMCA Lon 


Imported from CSV file 



G 

Distance Between WWMCA and Cloudsat Points (m) 


Imported from CSV file 



H 

Cloudsat Max Mask 


Imported from CSV file 



I 

Cloudsat Mask Text (20 threshold) 


Imported from CSV file 

Q 


J 

Cloudsat Occurrence Value (20 threshold) 


Imported from CSV file 

T5 


K 

Cloudsat Mask Text (30 threshold) 


Imported from CSV file 

!/3 

QJ 

U 


L 

Cloudsat Occurrence Value (30 threshold) 


Imported from CSV file 

O 

Ph 


M 

WWMCA Valid Time 


Imported from CSV file 



N 

WWMCA i 


Imported from CSV file 



O 

WWMCA j 


Imported from CSV file 



P 

Hemisphere 


Imported from CSV file 



Q 

WWMCA Layer 1 Cloud Cover (%) 


Imported from CSV file 



R 

WWMCA Layer 1 Cloud Type 


Imported from CSV file 



S 

WWMCA Layer 1 Height of Cloud Base (m) 


Imported from CSV file 



T 

WWMCA Layer 1 Height of Cloud Top (m) 


Imported from CSV file 



U 

WWMCA Layer 2 Cloud Cover (%) 


Imported from CSV file 



V 

WWMCA Layer 2 Cloud Type 


Imported from CSV file 
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Sheet 

Section 

Column 

Name 

Formula 

Remarks 



W 

WWMCA Layer 2 Height of Cloud Base (m) 


Imported from CSV file 



X 

WWMCA Layer 2 Height of Cloud Top (m) 


Imported from CSV file 



Y 

WWMCA Layer 3 Cloud Cover (%) 


Imported from CSV file 



Z 

WWMCA Layer 3 Cloud Type 


Imported from CSV file 



AA 

WWMCA Layer 3 Height of Cloud Base (m) 


Imported from CSV file 



AB 

WWMCA Layer 3 Height of Cloud Top (m) 


Imported from CSV file 

C3 

C3 


AC 

WWMCA Layer 4 Cloud Cover (%) 


Imported from CSV file 

— 

T5 


AD 

WWMCA Layer 4 Cloud Type 


Imported from CSV file 

0> 

cc 

5*5 


AE 

WWMCA Layer 4 Height of Cloud Base (m) 


Imported from CSV file 

u 

o 


AF 

WWMCA Layer 4 Height of Cloud Top (m) 


Imported from CSV file 

c- 


AG 

WWMCA Total Cloud Cover (%) 


Imported from CSV file 



AH 

WWMCA Cloud Cover Bin 


Imported from CSV file 



AI 

WWMCA Pixel Age (min) 


Imported from CSV file 



AJ 

Cloudsat Total Cloud Cover (% - 20 threshold) 


Imported from CSV file 



AK 

Cloudsat Cloud Eventing (20 threshold) 


Imported from CSV file 



AL 

Cloudsat Total Cloud Cover (% - 30 threshold) 


Imported from CSV file 



AM 

Cloudsat Cloud Eventing (30 threshold) 


Imported from CSV file 
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Sheet 

Section 

Column 

Name 

Formula 

Remarks 



A 

CSV Line Number 


Tracks CSV line numbers in this workbook 



B 

WWMCA valid time 

=’Processed Data'!A3 

From Processed Data sheet Column A 



C 

WWMCA latitude 

='Processed Data'!E3 

From Processed Data sheet Column E 



D 

WWMCA longitude 

='Processed Data'!F3 

From Processed Data sheet Column F 


CS 

a 

E 

WWMCA i 

='Processed Data'!N3 

From Processed Data sheet Column N 


F 

WWMCA j 

='Processed Data'!03 

From Processed Data sheet Column O 


Q 

'd 

G 

Cloudsat time 

='Processed Data'!B3 

From Processed Data sheet Column B 


QJ 

<*> 

Vl 

0> 

CJ 

o 

H 

WWMCA total cloud amount 

(%) 

='Processed Data'!AG3 

From Processed Data sheet Column AG 


s- 

CU 

I 

WWMCA cloud cover event 

='Processed Data'!AH3 

From Processed Data sheet Column AH 

C3 

£ 

o 

u 

ta 

J 

Cloudsat-20 total cloud 

amount (% ) 

='Processed Data'!AJ3 

From Processed Data sheet Column AJ 

"S 

— 


K 

Cloudsat-30 total cloud 

amount (%) 

='Processed Data'!AL3 

From Processed Data sheet Column AL 

s- 


L 

Cloudsat-20 cloud cover event 

='Processed Data'!AK3 

From Processed Data sheet Column AK 

E 


M 

Cloudsat-30 cloud cover event 

='Processed Data'!AM3 

From Processed Data sheet Column AM 



N 

Filtered WWMCA cloud cover 
event (Cloudsat-20) 

=IF(J3"n/a (less than 6 close points)", 

B) 

Filtered out WWMCA cloud cover bin values that 
corresponded to Cloudsat-20 cells that had "n/a 
(less than 6 close points)" and replace with "" 


d 

d 

O 

T3 

O 

Filtered WWMCA cloud cover 
event (Cloudsat-30) 

=IF(K3"n/a (less than 6 close points)", 
13) 

Filtered out WWMCA cloud cover bin values that 
corresponded to Cloudsat-30 cells that had "n/a 
(less than 6 close points)" and replace with "" 


0> 

a> 

P 

Filtered Cloudsat-20 total 

=IF(J3"n/a (less than 6 close points)", "", 

Filtered out the "n/a (less than 6 close points)" from 




cloud amount (%) 

J3) 

Cloudsat-20 total cloud amount and replace with "" 



n 

Filtered Cloudsat-30 total 

=IF(K3"n/a (less than 6 close points)", 

Filtered out the "n/a (less than 6 close points)" from 



V 

cloud amount (%) 

”",K3) 

Cloudsat-30 total cloud amount and replace with "" 



R 

Filtered Cloudsat-20 cloud 

cover event 

=IF(L3=-1, L3) 

Filtered out values of -1 from Cloudsat-20 cloud 

cover event 
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Sheet 

Section 

Column 

Name 

Formula 

Remarks 



S 

Filtered Cloudsat-30 cloud 

cover event 

=IF(M3=-1, M3) 

Filtered out values of -1 from Cloudsat-30 cloud 

cover event 




WWMCA cloud cover event 




A 

A 

T 

(Cloudsat-20) w/ values of '2' 

=IF(N3=2,N3) 




removed 




— 

T3 


WWMCA cloud cover event 




S- 

o> 

U 

(Cloudsat-30) w/ values of '2' 

=IF(03=2,03) 



- 4-1 


removed 





V 

Cloudsat-20 cloud cover event 
w/ values of '2' removed 

=IF(R3=2, R3) 




w 

Cloudsat-30 cloud coverevent 
w/ values of '2' removed 

=IF(S3=2,S3) 


eg 

S3 

T3 


%Diff in WWMCA & 



D 

O 

X 

Cloudsat-20 total cloud 

=(H3-P3)/H3 


Q 

G 


amount (%) 



CJ 

u 

•*-* 


%Diff in WWMCA & 



QJ 

o 

H 

Y 

Cloudsat-30 total cloud 

=(H3-Q3)/H3 


E 

© S3 


amount (%) 




u © 

g s 


Filtered % Diff in WWMCA & 


Filtered out values of ’#DIV/0!' from % diff in 


a> H 

SN «< 

z 

Cloudsat-20 total cloud 


WWMCA & Cloudsat-20 total cloud amount with a 


83 

2 


amount (%) 


blank by find/replace function. 


S3 


Filtered % Diff in WWMCA & 


Filtered out values of ’#DIV/0!' from % diff in 


u 

AA 

Cloudsat-30 total cloud 


WWMCA & Cloudsat-30 total cloud amount by 


— 


amount (%) 


find/replace function. 


A 

AZ 

Cloudsat time for Flagging 

=G3 

time in decimal form 


ie of D 
Flags 

BA 

Day/Night Flag 

=IF(AND(AZ3>0.625, AZ3<1), "Day", 
"Night") 

see Figure 12 for other daytime ranges 


S 

H 

BB 

Polar Night Flag 

IFfProcessed Data"!C3>=70, "Night", 
BA3) 

Used for Cloudsat latitudes above 70°N. For Polar 
Day, replace "Night" with "Day." 


80 






Sheet 

Section 

Column 

Name 

Formula 

Remarks 


3 ^ 

AB 

Event 3 (Cloudsat-20) Cell A 
(Hit) 

=IF(AND(N3=3, R3=3), "Hit", "") 




AC 

Event 3 (Cloudsat-20) Cell B 
(False Alarm) 

=IF(AND(N3=3, R3<2), "FA", "") 



su O 'S 
§ 

’■§ oU 

AD 

Event (Cloudsat-20) Cell C 
(Miss) 

=IF(AND(N3<2, R3=3), "Miss","") 




AE 

Event 3 (Cloudsat-20) Cell D 
(Correct Rejection) 

=IF(AND(N3<2, R3<2), "Reject", "") 


cs 

u 

<© 

AF 

Event 2 (Cloudsat-20) Cell A 
(Hit) 

=IF(AND(N3=2, R3=2), "Hit", "") 


a 

— 

A 

Q 

AG 

Event 2 (Cloudsat-20) Cell B 
(False Alarm) 

=IF(AND(N3=2, V3<3), "FA","") 


s- 

o> 

< 

Qm 

o 

AH 

Event 2 (Cloudsat-20) Cell C 
(Miss) 

=IF(AND(T3<3, R3=2), "Miss", "") 



t/J ® 

0) <N 

s ^ 

AI 

Event 2 (Cloudsat-20) Cell D 
(Correct Rejection) 

=IF(AND(T3<3, V3<3), "Reject", "") 



C/5 ^3 

0 i D 

3 3 

AJ 

Event 1 (Cloudsat-20) Cell A 
(Hit) 

=IF(AND(N3=1, R3=l), "Hit", "") 



H 

o 

S3 

a> 

AK 

Event 1 (Cloudsat-20) Cell B 
(False Alarm) 

=IF(AND(N3=1, R3>1), "FA", "") 



ox 

# g 

V-* 

S3 

AL 

Event 1 (Cloudsat-20) Cell C 
(Miss) 

=IF(AND(N3>1, R3=l), "Miss","") 



u 

AM 

Event 1 (Cloudsat-20) Cell D 
(Correct Rejection) 

=IF(AND(N3>1, R3>1), "Reject", "") 



81 













Sheet 

Section 

Column 

Name 

Formula 

Remarks 



AN 

Event 3 (Cloudsat-30) Cell A 
(Hit) 

=IF(AND(P33, S33), "Hit","") 



o 

rn 

AO 

Event 3 (Cloudsat-30) Cell B 
(False Alarm) 

=IF(AND(P33, S3<2), "FA", "") 



■m 

a 

v 

© 

o 

AP 

Event (Cloudsat-30) Cell C 
(Miss) 

=IF(AND(P3<2, S33), "Miss", "") 



U 

s- 

£ 

AQ 

Event 3 (Cloudsat-30) Cell D 
(Correct Rejection) 

=IF(AND(P3<2, S3<2), "Reject","") 


C3 

-m 

© 

P 

— 

< 

AR 

Event 2 (Cloudsat-30) Cell A 
(Hit) 

=IF(AND(P32, S32), "Hit","") 


TS 

© 

U 

© 

Cm 

O 

V 

© 

Sh 

AS 

Event 2 (Cloudsat-30) Cell B 
(False Alarm) 

=IF(AND(P32, W3<3), "FA","") 


a 

© 

© 

t/5 

_© 

AT 

Event 2 (Cloudsat-30) Cell C 
(Miss) 

=IF(AND(U3<3, S32), "Miss", "") 



»© 

© 

H 

AU 

Event 2 (Cloudsat-30) Cell D 
(Correct Rejection) 

=IF(AND(U3<3, W3<3), "Reject", "") 



a 

© 

ox 

AV 

for bin (1 -30) Cell A (Hit) 

=IF(AND(P31, S31), "Hit","") 



*-M 

© 

© 

U 

AW 

for bin (1 -30) Cell B (False 
Alarm) 

=IF(AND(P31, S3>1), "FA", "") 




AX 

for bin(l -30) Cell C (Miss) 

=IF(AND(P3>1, S31), "Miss", "") 




AY 

for bin (1 - 30) Cell D (Correct 
Rejection) 

=IF(AND(P3>1, S3>1), "Reject","") 
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Sheet 

Section 

Column 

Name 

Formula 

Remarks 



BC 

Event 3 (Cloudsat-20) Cell A 
(Hit) 

=IF($BB3”Day", AB3, "") 



o 

BD 

Event 3 (Cloudsat-20) Cell B 
(False Alarm) 

=IF($BB3”Day", AC3, "") 



1 

a 

n 

h3 

3 

BE 

Event (Cloudsat-20) Cell C 
(Miss) 

=IF($BB3"Day", AD3, "") 



o 

0 

£ 

BF 

Event 3 (Cloudsat-20) Cell D 
(Correct Rejection) 

=IF($BB3"Day", AE3, "") 


cs 

a 

Q 

s 

c« 

Q 

«w 

O 

BG 

BH 

Event 2 (Cloudsat-20) Cell A 
(Hit) 

Event 2 (Cloudsat-20) Cell B 
(False Alarm) 

=IF($BB3"Day", AEG, "") 

=IF($BB3”Day", AG3, "") 


u 

QJ 

E 

t/3 

© 

S- 

o 

o 

Ifl 

3 

cs 

H 

u 

fl 

© 

OX) 

# c 

a 

© 

BI 

BJ 

Event 2 (Cloudsat-20) Cell C 
(Miss) 

Event 2 (Cloudsat-20) Cell D 
(Correct Rejection) 

=IF($BB3”Day", AH3, "") 

=IF($BB3”Day", AI3, "") 



BK 

BL 

Event 1 (Cloudsat-20) Cell A 
(Hit) 

Event 1 (Cloudsat-20) Cell B 
(False Alarm) 

=IF($BB3”Day", AJ3, "") 

=IF($BB3”Day", AK3, "") 



u 

BM 

Event (Cloudsat-20) Cell C 
(Miss) 

=IF($BB3”Day", AL3, "”) 




BN 

Event 1 (Cloudsat-20) Cell D 
(Correct Rejection) 

=IF($BB3"Day", AM3, "") 
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Sheet 

Section 

Column 

Name 

Formula 

Remarks 



BO 

Event 3 (Cloudsat-30) Cell A 
(Hit) 

=IF($BB3”Day", AN3, '"') 



o 

rr> 

BP 

Event 3 (Cloudsat-30) Cell B 
(False Alarm) 

=IF($BB3”Day", A03, "") 



+* 

a 

on 

a 

BQ 

Event (Cloudsat-30) Cell C 
(Miss) 

=IF($BB3”Day", AP3, "") 



o 

0 

u 

© 

BR 

Event 3 (Cloudsat-30) Cell D 
(Correct Rejection) 

=IF($BB3"Day", AQ3, "") 



Cm 

QJ 

S 

*-m 

BS 

Event 2 (Cloudsat-30) Cell A 
(Hit) 

=IF($BB3"Day", AR3, "") 


a 

Q 

T3 

a 

Q 

®M 

© 

BT 

Event 2 (Cloudsat-30) Cell B 
(False Alarm) 

=IF($BB3"Day", AS3, "") 


0 > 

U 

QJ 

t« 

© 

S- 

© 

o 

BU 

Event 2 (Cloudsat-30) Cell C 
(Miss) 

=IF($BB3”Day", AT3,"") 



C/2 

3 

CS 

BV 

Event 2 (Cloudsat-30) Cell D 
(Correct Rejection) 

=IF($BB3"Day", AU3, ”") 



H 

© 

= 

BW 

Event 1 (Cloudsat-30) Cell A 
(Hit) 

=IF($BB3”Day", AV3, "") 



DU 

C 

*-m 

a 

BX 

Event 1 (Cloudsat-30) Cell B 
(False Alarm) 

=IF($BB3"Day", AW3, "") 



u 

BY 

Event 1 (Cloudsat-30) Cell C 
(Miss) 

=IF($BB3"Day", AX3, "") 




BZ 

Event 1 (Cloudsat-30) Cell D 
(Correct Rejection) 

=IF($BB3"Day", AY3, "") 
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Sheet 

Section 

Column 

Name 

Formula 

Remarks 



CA 

Event 3 (Cloudsat-20) Cell A 
(Hit) 

=IF($BB3"Night", AB3, "") 



o 

r* 

1 

CB 

Event 3 (Cloudsat-20) Cell B 
(False Alarm) 

=IF($BB3"Night", AC3, "") 



A 

C/2 

d 

s 

o 

CC 

Event (Cloudsat-20) Cell C 
(Miss) 

=IF($BB3"Night", AD3, "") 



u 

■- 

a 

CD 

Event 3 (Cloudsat-20) Cell D 
(Correct Rejection) 

=IF($BB3"Night", AE3, "") 



QJ 

S 

*-M 

■M 

CE 

Event 2 (Cloudsat-20) Cell A 
(Hit) 

=IF($BB3"Night", AF3, "") 


■M 

ft 

— 

d 

gf 

55 

CM 

CF 

Event 2 (Cloudsat-20) Cell B 
(False Alarm) 

=IF($BB3"Night”, AG3, "") 


U 

QJ 

•M 

a 

fa 

o 

1/2 

a> 

s- 

o 

QJ 

W 

QJ 

3 

ft 

H 

CJ 

S3 

CG 

CH 

Event 2 (Cloudsat-20) Cell C 
(Miss) 

Event 2 (Cloudsat-20) Cell D 
(Correct Rejection) 

=IF($BB3"Night", AH3,"") 

=IF($BB3"Night", AI3, "") 



Cl 

Event 1 (Cloudsat-20) Cell A 
(Hit) 

=IF($BB3"Night", AJ3, "") 



a> 

OX) 

# g 

fa 

fl 

CJ 

Event 1 (Cloudsat-20) Cell B 
(False Alarm) 

=IF($BB3"Night”, AK3, "") 



o 

U 

CK 

Event 1 (Cloudsat-20) Cell C 
(Miss) 

=IF($BB3"Night”, AL3, "") 




CL 

Event 1 (Cloudsat-20) Cell D 
(Correct Rejection) 

=IF($BB3"Night”, AM3, "") 
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Sheet 

Section 

Column 

Name 

Formula 

Remarks 



CM 

Event 3 (Cloudsat-30) Cell A 
(Hit) 

=IF($BB3"Night", AN3, "") 



o 

i 

CN 

Event 3 (Cloudsat-30) Cell B 
(False Alarm) 

=IF($BB3"Night", A03, "") 



S3 

Gfi 

s 

o 

CO 

Event (Cloudsat-30) Cell C 
(Miss) 

=IF($BB3"Night", AP3, "") 



0 

L. 

<2 

CP 

Event 3 (Cloudsat-30) Cell D 
(Correct Rejection) 

=IF($BB3"Night", AQ3, "") 



0 > 

s 

*-M 

■M 

CQ 

Event 2 (Cloudsat-30) Cell A 
(Hit) 

=IF($BB3"Night", AR3, "") 


■M 

ft 

— 

T3 

gf 

2 

CM 

CR 

Event 2 (Cloudsat-30) Cell B 
(False Alarm) 

=IF($BB3"Night”, AS3, "") 


U 

QJ 

•M 

a 

fa 

o 

1/3 

a> 

s- 

o 

u 

w 

QJ 

3 

ft 

H 

CJ 

S3 

CS 

CT 

Event 2 (Cloudsat-30) Cell C 
(Miss) 

Event 2 (Cloudsat-30) Cell D 
(Correct Rejection) 

=IF($BB3"Night", AT3, ”") 

=IF($BB3"Night", AU3, "") 



CU 

Event 1 (Cloudsat-30) Cell A 
(Hit) 

=IF($BB3"Night", AV3, "") 



a> 

OX) 

# g 

*M 

fi 

CV 

Event 1 (Cloudsat-30) Cell B 
(False Alarm) 

=IF($BB3"Night”, AW3, "") 



o 

U 

CW 

Event 1 (Cloudsat-30) Cell C 
(Miss) 

=IF($BB3"Night”, AX3, "") 




CX 

Event 1 (Cloudsat-30) Cell D 
(Correct Rejection) 

=IF($BB3"Night”, AY3, "") 
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Numberof Occurrences Numberof Occurrences Numberof Occurrences 


APPENDIX C. RESULTS FOR WWMCA BOXES 22 AND 29 FOR 
CLOUDSAT CLOUD MASK VALUE OF 20 USED FOR CLOUD/NO¬ 
CLOUD THRESHOLD 


A. WWMCA BOX 22 


January Daytime Occurrences WWMCA (22) - Cloudsat (20) 2010 


April Daytime Occurrences WWMCA (22)- Cloudsat (20) 2010 



A B C D 


■ Cloud 

■ ProbableCloud 

■ No Cloud 


60000 



■ Cloud 

■ ProbableCloud 

■ No Cloud 


January Nighttime Occurrences WWMCA (22) - Cloudsat (20) 2010 


April Nighttime Occurrences WWMCA (22) - Cloudsat (20) 2010 



A B C D 


■ Cloud 

■ ProbableCloud 

■ No Cloud 


60000 



■ Cloud 

■ ProbableCloud 

■ No Cloud 


January All Day Occurrences WWMCA (22) - Cloudsat (20) 2010 



A B C D 


April All Day Occurrences WWMCA (22) - Cloudsat (20) 2010 


120000 



■ Cloud 

■ ProbableCloud 

■ No Cloud 


Figure 31. Number of occurrences in January and April 2010 for WWMCA Box 22 for 
all three cloud cover events (cloud, probable cloud, and no-cloud) using a 
Cloudsat cloud mask value of 20 for cloud/no-cloud threshold. The “cloud” 
event (blue) represents the “definite cloud” event. The horizontal axes are the 
occurrence categories: A—Hits; B—False Alarms; C—Misses; and D— 
Correct Rejections. The top panels are the occurrences for daytime, the 
middle panels are nighttime occurrences, and the bottom panels are the 
combined results of both daytime and nighttime occurrences. 
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Numberof Occurrences Numberof Occurrences Numberof Occurrences 


July Daytime Occurences WWMCA (22) - Cloudsat (20) 2010 



■ ProbableCloud 


October Daytime Occurrences WWMCA (22) - Cloudsat (20) 2010 
60000 



■ Pro table Cloud 


July Nightime Occurrences WWMCA (22) - Cloudsat (20) 2010 



■ Cloud 

■ Probable Cloud 

■ No Cloud 


October Nighttime Occurrences WWMCA (22) - Cloudsat (20) 2010 
60000 



■ ProbableCloud 



October All Day Occurrences WWMCA (22) - Cloudsay (20) 2010 



■ Cloud 

■ ProbableCloud 

■ No Cloud 


Figure 32. Number of 2 x 2 contingency table occurrences in July and October 2010 for 
WWMCA Box 22 for all three cloud cover events: Definite-Cloud (blue), 
Probable-Cloud (red), and No-Cloud (green). On the horizontal axes: A 
indicates Hits, B indicates False Alarms, C indicates Misses, and D indicates 
Correct Rejections. The top panels shows the daytime occurrences, the 
middle panels shows the nighttime occurrences, and the bottom panels show 
the results for both daytime and nighttime occurrences. Results based on 
using a Cloudsat cloud mask value of 20 for the cloud/no-cloud threshold. 
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Daytime Marginal Distributions of Cloud Cover Events for WWMCA 
(22) - Cloudsat (20) 2010 
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All Day Marginal Distributions of Cloud Cover Events for WWMCA 
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Figure 33. Marginal distributions of cloud cover events for WWMCA and Cloudsat for 
WWMCA Box 22: Definite-Cloud, Probable-Cloud, and No-Cloud). The 
calculation of the marginal distributions is shown in Table 7. The top panel 
shows the results for daytime data, the middle panel for nighttime data, and 
the bottom panel for both daytime and nighttime data. Results based on using 
a Cloudsat cloud mask value of 20 for the cloud/no-cloud threshold. 
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Daytime Hit Rate WWMCA (22) - Cloudsat (20) 2010 
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Figure 34. WWMCA probability of detection (POD), or hit rate (H), for WWMCA Box 
22 for all three cloud cover events: Definite-Cloud (blue), Probable-Cloud 
(red), and No-Cloud (green). The top panel shows the results for daytime 
data, the middle panel for nighttime data, and the bottom panel for both 
daytime and nighttime data. Results based on using a Cloudsat cloud mask 
value of 20 for the cloud/no-cloud threshold. 
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Figure 35. WWMCA false alarm ratios for WWMCA Box 22 for all three cloud cover 
events: Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud (green). 

The top panel shows the results for daytime data, the middle panel for 
nighttime data, and the bottom panel for both daytime and nighttime data. 
Results based on using a Cloudsat cloud mask value of 20 for the cloud/no¬ 
cloud threshold. 
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Figure 36. WWMCA proportion correct (PC) for WWMCA Box 22 for all three cloud 
cover events: Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud 
(green). The top panel shows the results for daytime data, the middle panel 
for nighttime data, and the bottom panel for both daytime and nighttime data. 
Results based on using a Cloudsat cloud mask value of 20 for the cloud/no¬ 
cloud threshold. 
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Figure 37. WWMCA threat scores (TS) for WWMCA Box 22 for all three cloud cover 
events: Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud (green). 

The top panel shows the results for daytime data, the middle panel for 
nighttime data, and the bottom panel for both daytime and nighttime data. 
Results based on using a Cloudsat cloud mask value of 20 for the cloud/no¬ 
cloud threshold. 
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Daytime Bias WWMCA (22) - Cloudsat (20) 2010 
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Figure 38. WWMCA bias (B) for WWMCA Box 22 for all three cloud cover events: 

Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud (green). The top 
panel shows the results for daytime data, the middle panel for nighttime data, 
and the bottom panel for both daytime and nighttime data. The heavy dark 
line at B = 1 indicates an unbiased analysis. Values greater (less) than 1 
indicate a cloud cover event that was over (under) analyzed. Results based on 
using a Cloudsat cloud mask value of 20 for the cloud/no-cloud threshold. 
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Figure 39. WWMCA Heidke skill score (HSS) for WWMCA Box 22 for all three cloud 
cover events: Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud 
(green). The top panel shows the results for daytime data, the middle panel 
for nighttime data, and the bottom panel for both daytime and nighttime data. 

The heavy dark line at the zero value indicates that the WWMCA is 
equivalent to reference analyses (random analyses statistically independent of 
observations). A perfect WWMCA cloud analysis would have a value of one. 
Negative values indicate WWMCA performs worse than reference analyses. 
Results based on using a Cloudsat cloud mask value of 20 for the cloud/no¬ 
cloud threshold. 


95 



































Numberof Occurrences Numberof Occurrences Numberof Occurrences 


B. WWMCA BOX 29 


January Daytime Occurrences WWMCA (29) - Cloudsat (20) 2010 


April Daytime Occurrences WWMCA (29)- Cloudsat (20) 2010 



A B C D 


80000 

70000 

g 60000 

c 

a 

^ 50000 
40000 

O 

jj 30000 

E 

| 20000 
10000 
0 



A B C D 


■ Cloud 

■ ProbableCloud 

■ No Cloud 


January Nighttime Occurrences WWMCA (29) - Cloudsat (20) 2010 



■ Cloud 

■ ProbableCloud 

■ No Cloud 



A B C D 


January All Day Occurrences WWMCA (29) - Cloudsat (20) 2010 


April All Day Occurrences WWMCA (29) - CJoudsat (320) 2010 



i ProbableCloud 


8 300000 



■ ProbableCloud 


Figure 40. Number of 2 x 2 contingency table occurrences in January and April 2010 for 
WWMCA Box 29 for all three cloud cover events: Definite-Cloud (blue), 
Probable-Cloud (red), and No-Cloud (green). On the horizontal axes: A 
indicates Hits, B indicates False Alarms, C indicates Misses, and D indicates 
Correct Rejections. The top panels shows the daytime occurrences, the 
middle panels shows the nighttime occurrences, and the bottom panels show 
the results for both daytime and nighttime occurrences. Results based on 
using a Cloudsat cloud mask value of 20 for the cloud/no-cloud threshold. 
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Figure 41. Number of 2 x 2 contingency table occurrences in July and October 2010 for 
WWMCA Box 29 for all three cloud cover events: Definite-Cloud (blue), 
Probable-Cloud (red), and No-Cloud (green). On the horizontal axes: A 
indicates Hits, B indicates False Alarms, C indicates Misses, and D indicates 
Correct Rejections. The top panels shows the daytime occurrences, the 
middle panels shows the nighttime occurrences, and the bottom panels show 
the results for both daytime and nighttime occurrences. Results based on 
using a Cloudsat cloud mask value of 20 for the cloud/no-cloud threshold. 
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Daytime Probability of Events WWMCA (29) - Cloudsat (20) 2010 
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Figure 42. Marginal distributions of cloud cover events for WWMCA and Cloudsat for 
WWMCA Box 29: Definite-Cloud, Probable-Cloud, and No-Cloud). The 
calculation of the marginal distributions is shown in Table 7. The top panel 
shows the results for daytime data, the middle panel for nighttime data, and 
the bottom panel for both daytime and nighttime data. Results based on using 
a Cloudsat cloud mask value of 20 for the cloud/no-cloud threshold. 
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Figure 43. WWMCA probability of detection (POD), or hit rate (H), for WWMCA Box 
29 for all three cloud cover events: Definite-Cloud (blue), Probable-Cloud 
(red), and No-Cloud (green). The top panel shows the results for daytime 
data, the middle panel for nighttime data, and the bottom panel for both 
daytime and nighttime data. Results based on using a Cloudsat cloud mask 
value of 20 for the cloud/no-cloud threshold. 
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Figure 44. WWMCA false alarm ratios for WWMCA Box 29 for all three cloud cover 
events: Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud (green). 

The top panel shows the results for daytime data, the middle panel for 
nighttime data, and the bottom panel for both daytime and nighttime data. 
Results based on using a Cloudsat cloud mask value of 20 for the cloud/no¬ 
cloud threshold. 
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Figure 45. WWMCA proportion correct (PC) for WWMCA Box 29 for all three cloud 
cover events: Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud 
(green). The top panel shows the results for daytime data, the middle panel 
for nighttime data, and the bottom panel for both daytime and nighttime data. 
Results based on using a Cloudsat cloud mask value of 20 for the cloud/no¬ 
cloud threshold. 
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Figure 46. WWMCA threat scores (TS) for WWMCA Box 29 for all three cloud cover 
events: Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud (green). 

The top panel shows the results for daytime data, the middle panel for 
nighttime data, and the bottom panel for both daytime and nighttime data. 
Results based on using a Cloudsat cloud mask value of 20 for the cloud/no¬ 
cloud threshold. 
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Figure 47. WWMCA bias (B) for WWMCA Box 29 for all three cloud cover events: 

Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud (green). The top 
panel shows the results for daytime data, the middle panel for nighttime data, 
and the bottom panel for both daytime and nighttime data. The heavy dark 
line at B = 1 indicates an unbiased analysis. Values greater (less) than 1 
indicate a cloud cover event that was over (under) analyzed. Results based on 
using a Cloudsat cloud mask value of 20 for the cloud/no-cloud threshold. 
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Figure 48. WWMCA Heidke skill score (HSS) for WWMCA Box 29 for all three cloud 
cover events: Definite-Cloud (blue), Probable-Cloud (red), and No-Cloud 
(green). The top panel shows the results for daytime data, the middle panel 
for nighttime data, and the bottom panel for both daytime and nighttime data. 

The heavy dark line at the zero value indicates that the WWMCA is 
equivalent to reference analyses (random analyses statistically independent of 
observations). A perfect WWMCA cloud analysis would have a value of one. 
Negative values indicate WWMCA performs worse than reference analyses. 
Results based on using a Cloudsat cloud mask value of 20 for the cloud/no¬ 
cloud threshold. 
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